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ABSTRACT 


In  an  environment  of  scarce  resources  and  rising  federal  deficits  the  people  not 
only  expect,  but  demand  greater  accountability  for  the  spending  of  public  funds.  This 
demand  has  created  a  trend  in  the  public  sector,  not  only  in  the  United  States,  but 
worldwide  as  well,  towards  the  importation  of  private  sector  business  practices  to 
improve  accountability-oriented  analysis.  One  example  is  increased  emphasis  on  return 
on  investment  (ROI)  analysis  in  public  sector  organizations. 

Development  and  application  of  ROI  analysis  is  challenging  in  the  public  sector 
since  most  government  organizations  do  not  generate  profit  necessary  for  calculation  of 
ROI  in  the  manner  in  which  it  is  done  in  the  private  sector.  This  thesis  develops  the 
methodology  necessary  for  use  of  ROI  analysis  in  the  public  sector.  ROI  methodology  is 
applied  for  test  evaluation  with  the  Space  and  Naval  Warfare  Systems  Command 
(SPAWAR)  in  San  Diego.  The  test  demonstrates  that  ROI  can  be  applied  successfully  to 
assess  the  relative  efficiency  of  value-added  work  and  to  improve  the  process  of  choosing 
between  investment  alternatives.  Properly  designed  ROI  analysis  reveals  how  and  for 
what  goods  and  services  money  is  spent  and  provides  a  means  for  comparing  the  value 
derived  from  investment  and  work  performed. 
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I.  INTRODUCTION 


A.  PURPOSE 

This  thesis  identities  and  assesses  several  previous  efforts  to  use  return  on 
investment  (ROI)  criteria  as  benchmarks  in  the  public  sector.  Lessons  learned  from  this 
review  have  been  applied  in  the  development  of  a  notional  ROI  (NROI)  fonnula 
identified  for  applicability  with  the  Program  Executive  Office  (PEO)  Command,  Control, 
Communication,  Computers,  and  Intelligence  (C4I)  Shipbuilding  Coordinator  (SC)  at 
Space  and  Naval  Warfare  Systems  Command  (SPAWAR).  Specifically,  the  data  used  in 
the  NROI  fonnula  apply  to  the  design,  procurement,  testing,  installation,  integration, 
personnel  training,  and  certification  of  the  radio  control  suite  (RCS)  equipment  and 
associated  gear  for  the  first  ship  of  T-AKE  new  construction.  The  same  formula  was 
applied  for  comparison  purposes  with  a  similar  effort  on  the  LPD-19,  third  ship  of  the 
LPD-17  class.  This  thesis  provides  a  detailed  analysis  of  the  strengths  and  weaknesses  of 
applying  ROI  in  the  public  sector,  with  specific  emphasis  on  original  work  performed 
cooperatively  with  PEO  C4I  SC. 

B.  BACKGROUND 

The  Navy  and  Department  of  Defense  (DoD)  are  attempting  to  define  and  apply 
improved  methods  for  evaluating  ROI  in  various  ways,  under  pressure  from  Congress, 
the  President's  Office  of  Management  and  Budget  and  the  scarcity  of  resources  available 
for  defense  (McCaffery  and  Jones,  2004:  1-20).  ROI  is  one  of  the  key  methods  used  to 
quantify  the  level  of  success  achieved  or  achievable  in  a  business  endeavor.  The  concept 
of  ROI  is  used  throughout  private  industry  not  only  to  detennine  past  results,  but  also  to 
evaluate  the  current  situation  and  as  a  decision  making  tool  for  the  future.  The 
advantages  of  ROI  are  clear  in  that  it  provides  the  flexibility  to  anticipate  output  changes 
in  advance.  This  benefit  results  in  the  ability  to  not  only  preview  the  future  in  a  real 
world  sense,  but  also  to  modify  the  inputs  to  the  numerator  and  denominator  of  the 
equation  to  model  potential  courses  of  action  for  the  enterprise. 


1 


Although  a  useful  concept,  ROI  does  not  easily  transition  for  use  in  the  public 
sector.  Unlike  private  enterprise,  the  public  sector  has  no  “profit”  or  “total  sales”  to  use 
in  the  equation.  The  increasing  need  for  some  method  to  quantify  ROI  in  the  public 
arena  has  led  to  multiple  attempts  from  a  diverse  group  of  public  enterprises  with  varying 
results.  The  Australian  government  placed  increased  emphasis  on  what  they  tenned  the 
“value  added”  approach  in  an  effort  to  detennine  the  output  they  were  receiving  as  a 
result  of  budgetary  expenditures.  The  Royal  New  Zealand  Navy  desired  a  determination 
of  ROI  for  the  implementation  of  a  retention  bonus  plan  used  to  control  the  attrition 
problem  that  was  being  experienced  with  marine  engineers.  Both  of  these  results  were 
somewhat  mixed  with  valuable  lessons  learned.  The  United  States  Postal  Service  (USPS) 
met  with  a  greater  level  of  success  in  their  effort,  due  largely  to  the  fact  that  they  are  run 
much  more  like  a  private  enterprise.  Although  not  seeking  to  be  “profitable,”  the  USPS 
does  generate  revenue  which  can  be  used  in  the  numerator  of  the  formula  which  when 
divided  by  the  USPS  asset  base  in  the  denominator  results  in  a  fairly  conventional  ROI. 
Finally,  the  US  Navy  Dental  community  effort  was  much  more  ambitious  in  that  it 
attempted  to  convert  non  cash  outputs  into  cash  equivalents  in  order  to  closely  adhere  to 
the  traditional  ROI  fonnula.  The  resulting  Navy  Dental  ROI  was  dogged  by  the 
questionable  accuracy  of  some  inputs,  but  the  overall  approach  remained  fundamentally 
sound. 

The  previous  efforts  focused  on  in  this  study  have  helped  define  the  need  for  a 
new  method  to  detennine  ROI  in  public  sector  enterprises.  The  intent  of  this  thesis  is  to 
review  the  previous  efforts  and  to  develop  a  new  approach  for  attaining  this  important 
goal  as  well  as  to  provide  a  workable  response  to  the  Chief  of  Naval  Operation’s  (CNO) 
call  for  an  innovative  method  of  linking  investments  with  measurable  outputs.  In 
November  of  2003,  in  an  Echelon  II  visit  feedback  memo,  the  CNO  Admiral  Vern  Clark 
outlined  the  need  to  develop  a  workable  ROI  formula: 

This  is  an  area  where  we  are  struggling  the  most.  We  need  to  know  that 
we  are  making  the  right  type  and  level  of  investment.  We  have  made 
some  progress  in  understanding  what  we  are  investing  in  and  have  even 
made  progress  in  understanding  the  output  of  our  processes.  However,  we 
are  struggling  to  link  the  two  (investment  to  output).  We  need  to  model 
how  increases  or  decreases  in  investments  (people,  dollars,  and 
technology)  will  change  the  output. 
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It  is  with  the  CNO’s  request  in  mind  that  the  PEO  C4I  SC  T-AKE  ROI  project 
was  initiated.  The  design,  installation  and  integration  of  the  T-AKE  class  command  and 
control  suite  equipment  —  specifically  the  Radio  Control  Suite  (RCS)  —  including 
ancillary  equipment  costs  is  used  as  the  test  platform  to  apply  what  we  term  the  NROI 
method  specifically  applicable  for  use  in  the  public  sector.  Our  definition  of  NROI  is 
based  on  the  asset  turnover  ratio  and  return  on  asset  approach  as  they  are  referred  to  in 
private  sector  accounting  and  finance. 


C.  RESEARCH  OBJECTIVES 

The  immediate  objective  of  this  effort  is  to  develop  a  workable  NROI  formula  for 
use  within  the  SPA  WAR  organization,  specifically  PEO  C4I  SC.  In  a  more  broad  sense, 
however,  the  underlying  objective  is  to  validate  an  approach  to  developing  NROI 
formulae  for  use  in  the  public  sector  across  a  wide  variety  of  organizations.  The 
sequence  of  development  of  the  NROI  formula  is  reported  in  this  thesis  to  provide  the 
reader  with  a  road  map  to  follow,  including  valuable  examples  and  lessons  learned. 


D.  RESEARCH  METHODOLOGY 

The  methodological  approach  used  in  this  thesis  evolved  as  the  research  project 
and  process  unfolded  over  a  period  of  approximately  ten  months.  The  initial  phase 
included  extensive  (literally  worldwide)  research  of  past  efforts  at  determining  ROI  in  the 
public  sector.  After  deriving  a  number  of  lessons  learned  from  these  past  efforts,  focus 
shifted  to  a  determination  of  the  criteria  that  had  to  be  followed  for  successful  NROI 
development.  This  step  was  followed  by  a  series  of  assessments  of  the  NROI  formula 
under  development  for  the  T-AKE  project,  including  the  necessary  reduction  of  scope  to 
a  focus  on  solely  the  RCS.  The  ensuing  stages  of  formula  development  and  evolution 
incorporated  elements  from  the  research  on  what  others  had  attempted  (in  the  private  and 
public  sectors)  and  what  was  necessary  for  design  and  use  with  the  specific  units  of 
analysis  (Navy  ships)  under  study.  Finally,  the  NROI  fonnula  was  used  to  compare 
return  on  investment  in  the  RCS  for  the  first  T-AKE  class  ship  in  production  at  different 
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points  in  time,  and  the  RCS  for  both  T-AKE  1  and  LPD-19.  More  detailed  articulation  of 
the  development  of  the  NROI  methodology  follows. 
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II.  TRADITIONAL  VS.  NOTIONAL  ROI 


A.  TRADITIONAL  ROI 

1.  How  to  Determine  Traditional  ROI 

ROI  has  traditionally  been  measured  in  the  private  sector  to  quantify  an 

organization’s  past,  present,  and  potential  future  performance.  There  are  several  methods 

by  which  an  organization  can  determine  its  ROI.  Most  compare  the  net  financial  output 

of  a  company,  or  profit,  to  the  financial  input.  One  of  the  most  common  methods  is  to 

compute  a  percentage  return  on  a  company’s  assets.  An  organization  can  determine  how 

efficiently  it  has  used  its  assets  by  comparing  a  period’s  operating  income  to  the  total 

amount  the  company  has  invested  in  the  assets  that  produced  that  income.  ROI  is 

traditionally  calculated  as  follows  (Garrison  and  Noreen,  2003:  542): 

„  ,  Net  Operating  Income 

Return  on  Investment  = - 

A  verage  Operating  Assets 

Net  operating  income  is  the  difference  between  revenue  and  expenses,  usually 
before  taxes  and  interest.  An  average  asset  base  is  normally  used  since  the  amount  of 
assets  in  use  may  have  changed  during  the  period  of  measurement.  Regardless  of  the 
exact  method  of  measurement,  a  higher  return  indicates  a  more  proficient  use  of 
organizational  assets  and  ultimately  a  higher  return  for  its  shareholders. 

2.  Practical  Applications  of  Traditional  ROI 

ROI  calculations  also  may  be  used  to  determine  the  potential  reward  of  a  single 
investment  decision  or  to  assist  in  choosing  between  multiple  investment  options.  For  a 
single  investment  decision,  forecasted  streams  of  revenue  are  estimated  and  compared  to 
the  expected  capital  investment  and  operating  costs.  These  are  compared  over  the  life  of 
the  proposed  project  and  used  to  determine  an  internal  rate  of  return  (IRR),  actually  a 
forecasted  ROI.  The  IRR  is  then  compared  to  a  firm’s  cost  of  capital  for  a  single 
investment  decision.  It  can  also  be  compared  to  the  IRR  forecasted  from  other 
investment  decisions  in  the  case  of  multiple  options  to  assist  in  choosing  between  them. 
With  reasonable  forecasting  accuracy,  this  becomes  an  effective  tool  used  in  the  private 
sector  for  deciding  between  capital  venture  decisions. 
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3.  Traditional  ROI  Example 

There  are  two  frequently  used  methods  to  determine  a  corporation’s  ROI. 
Consider  an  investor  deciding  whether  or  not  to  make  an  investment  in  PepsiCo  in  2002. 
One  method  for  estimating  PepsiCo’s  future  performance  is  to  look  at  its  previous  year’s 
use  of  assets.  This  is  commonly  referred  to  as  an  organization’s  return  on  assets,  or 
ROA.  This  was  the  method  discussed  in  the  previous  section.  Selected  data  for  PepsiCo 
taken  from  2001  (Brealey,  Marcus,  and  Myers,  2004:  450-51)  are  presented  in  Table  1. 
Table  1.  Selected  Financial  Data  for  PepsiCo:  2001  (millions  of  dollars) 


Earnings  Before  Interest  and  Taxes 

$4,181 

Net  Income  (after  interest  and  taxes) 

2,662 

Total  Assets 

21,695 

Total  Shareholder  Equity 

8,648 

To  determine  PepsiCo’s  ROA  for  2001,  their  earnings  before  interest  and  taxes 
are  taken  from  the  income  statement  and  must  be  divided  by  their  total  assets  from  the 
balance  sheet.  The  result  is  then  multiplied  by  one  hundred  to  produce  a  percentage 
ROA.  Thus,  for  PepsiCo  in  2001; 

IP  A  OTD 

ROA  (%)  =  — - X 1 00  =  1 9.3% 

$21. 7B 

This  indicates  that  every  dollar  invested  in  assets  at  PepsiCo  yielded  19.3  cents  of  return 
in  2001. 

Another  more  relevant  method  to  the  investor  would  be  to  determine  PepsiCo’s 
return  on  equity,  or  ROE  for  2001.  This  explicitly  gives  the  return  on  investor  equity  in 
PepsiCo.  ROE  is  determined  by  dividing  the  net  income  (income  after  interest  and  taxes) 
by  the  corporation’s  total  shareholder  equity.  The  result  is  then  multiplied  by  one  hundred 
to  produce  a  percentage  ROE.  Thus  for  PepsiCo  in  2001; 

ROE  (%)  =  X 1 00  =  3 1 .4% 

$8.6B 

This  indicates  that  every  dollar  invested  in  PepsiCo  by  investors  yielded  31.4  cents  of 
return  to  the  shareholders  in  2001. 
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It  is  important  to  keep  in  mind  that  taken  on  its  own  the  ROI  is  of  limited  value. 
In  this  example  it  would  be  wise  to  compare  the  ROA  and  ROE  for  PepsiCo  to  prior 
years  or  with  other  companies  in  the  same  business  during  the  same  year.  This 
comparability  is  very  helpful  in  determining  if  the  ROI  is  superior,  average,  or  mediocre. 
Careful  evaluation  of  the  inputs  to  the  ROI  formula  can  uncover  what  may  be  the  root  of 
the  success  or  problem. 

4.  Difficulties  Utilizing  Traditional  ROI  in  the  Public  Sector 

Public  sector  ROI  calculations  are  considerably  more  problematic  to  utilize  than 
in  private  industry.  The  traditional  method  of  detennining  investment  returns  in  the 
private  sector  is  not  directly  compatible  with  many  public  sector  organizations.  Consider 
how  a  public  sector  organization  would  detennine  its  financial  output.  Many  public 
sector  organizations  do  not  produce  revenues  or  generate  profits  as  outputs.  Therefore, 
their  outputs  are  difficult  to  quantify  in  dollars.  Instead,  they  provide  a  service  or 
capability  to  the  public.  Oftentimes  this  service  or  capability  is  unique  to  the  public 
sector  and  is  not  produced  by  the  private  sector.  This  increases  the  difficulty  when  trying 
to  value  these  unique  services  or  capabilities.  For  example,  how  much  value  is  added  to 
the  respective  service  when  another  tank  or  fighter  jet  is  produced?  We  certainly  know 
what  they  cost.  However,  it  is  difficult  to  quantify  their  value  added  to  the  Anny  or  Air 
Force.  Placing  dollar  values  on  these  items  is  complex  since  similar  items  are  not  valued 
in  the  private  sector.  The  value  added  to  the  services  from  these  items  cannot  be  easily,  if 
at  all,  measured  in  dollars.  This  makes  the  use  of  traditional  ROI  criteria  impossible. 

Some  public  sector  organizations  could  be  measured  by  the  equivalent  value  of 
the  service  or  capability  provided  in  the  private  sector.  For  example,  a  comparison  could 
be  made  between  the  USPS  and  the  United  Parcel  Service.  Perhaps  a  cost  comparison  for 
compatible  services  between  private  and  public  sector  organizations  could  be  used  to 
measure  performance.  However,  many  public  sector  companies  do  not  have  comparable 
organizations  in  the  private  sector.  For  example,  consider  the  DoD.  The  DoD  provides 
defensive  and  offensive  capability  for  the  United  States.  This  capability  cannot  be 
measured  against  the  private  sector  due  to  the  uniqueness  of  the  services  it  provides. 
Therefore,  to  facilitate  a  ROI  metric  for  many  public  sector  organizations,  a  different 
approach  needs  to  be  used. 
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B.  NOTIONAL  ROI 

1.  Exploring  ROI  in  the  Public  Sector 

ROI  measurements  are  under  exploration  in  the  public  sector  for  three  primary 
reasons.  First,  as  with  any  private  sector  corporation,  there  are  significantly  more 
investment  opportunities  than  public  funds  available.  There  is  intense  competitive 
pressure  between  organizations  to  continually  prove  their  need  for  additional  or  even 
continued  program  funding.  Deciding  between  these  alternatives  is  oftentimes  subjective 
in  nature  since  objective  data  is  not  available.  Realistically,  some  public  organizations 
will  be  funded  regardless  of  their  ROI.  However,  ROI  measurements  could  provide  one 
metric  to  objectively  decide  between  investment  alternatives  in  public  programs.  They 
could  also  be  used  by  organizations  to  show  their  value  added  to  the  public,  and 
consequently  provide  support  for  their  continued  funding. 

Second,  increased  public  spending  and  rising  budget  deficits  have  considerably 
raised  the  public’s  concern  for  the  way  the  public  sector  spends  its  money.  There  has 
been  a  notable  increase  in  the  required  accountability  of  the  public  sector  to  the 
taxpayers.  Evidence  of  this  is  the  Government  Performance  and  Results  Act  of  1993. 
The  general  purpose  of  this  legislation  is  to  establish  metrics  within  the  United  States 
government  to  hold  organizations  accountable  (Phillips  and  Phillips,  2002:  4).  ROI 
measurements  are  one  way  this  accountability  requirement  to  taxpayers  can  be  satisfied. 

Finally,  there  has  been  a  long  and  continuing  trend  within  the  public  sector  to 
import  successful  business  practices  from  the  private  sector.  This  is  no  surprise  since 
many  of  the  public  sector’s  leaders  had  previous  careers  in  the  private  sector.  In  fact, 
three  of  the  public  sector’s  most  senior  leaders,  the  current  United  States  President,  Vice 
President,  and  the  Secretary  of  Defense,  are  all  previous  chief  executive  officers  from  the 
business  community.  In  addition,  the  private  sector  is  generally  viewed  by  the  American 
public  and  many  academics  as  more  efficient  than  the  public  sector.  Unless  a  private 
organization  produces  a  unique  product  or  service,  efficiency  is  essential  for  competitive 
survival.  These  same  pressures  are  not  present  in  the  public  sector  to  the  same  degree 
and  facilitate  the  increased  efficiency  in  the  private  sector.  Consequently,  the  public 
sector  oftentimes  looks  to  mimic  the  more  efficient  practices  in  the  private  sector.  ROI 
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measurement  is  just  one  of  many  business  practices  within  the  private  sector  that  is  under 
experimentation  in  the  public  sector. 

2.  Non-Monetized  Notional  ROI 

The  inability  to  directly  apply  ROI  techniques  to  many  organizations  in  the  public 
sector  indicates  that  a  different  approach  is  required.  One  such  approach  includes  the  use 
of  cost  effectiveness  analysis  to  provide  a  useful  framework  with  which  to  assign  weights 
to  the  numerator  variables.  In  Cost  Benefit  Analysis:  Concepts  and  Practice,  the  authors 
address  this  issue: 

If  the  analyst  is  unable... to  monetize  the  major  benefit,  then  cost- 
effectiveness  analysis  may  be  appropriate.  Because  not  all  of  the  impacts 
can  be  monetized,  it  is  not  possible  to  estimate  net  benefits.  The  analyst 
can,  however,  construct  a  ratio  involving  the  quantitative,  but  non- 
monetized,  benefit  and  total  dollar  costs  (Boardman,  et  ah,  1996:  36). 

This  was  the  initial  approach  used  for  analyzing  the  PEO  C4I  SC  T-AKE  project.  The 
resulting  formula  would  produce  a  non-monetary  output  considered  to  be  the  NROI.  As 
changes  to  the  numerator  variables  or  the  denominator  are  considered,  a  new  NROI  can 
be  generated  to  determine  the  potential  impact  of  the  changes. 

In  order  for  the  NROI  formula  to  be  of  credible  value,  weights  must  be  assigned 
to  each  of  the  numerator  variables.  Weights  are  indicative  of  how  the  decision  makers 
prefer  to  balance  the  impact  of  the  attributes.  This  step  is  extremely  important  since  the 
weight  distribution  has  a  tremendous  impact  on  the  output.  Determination  of  weights  can 
be  an  objective  result  of  models  and  data  analysis,  a  subjective  result  of  discussion  by  the 
decision  makers,  or  a  combination  of  both.  There  are  four  common  methods  for 
determining  weights:  equal  weighting,  rank  reciprocal,  pair-wise  comparison,  and  direct 
assessment  (McNab,  2004).  Table  2  presents  a  comparison  of  the  various  methods  using 
the  six  categories  from  the  initial  NROI  formula  for  the  T-AKE  project  as  an  example. 
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Table  2.  Weighting  Method  Comparison 

Weighting  Methods 


Effectiveness 

Equal 

Rank 

Pair-wise 

Direct 

Weighting 

Reciprocal 

Comparison 

Assessment 

CP 

_  Capability 

.167 

1/1  ->60/147 

9->9/65 

.10 

CT 

-  Cost 

.167 

1/2 ->30/1 47 

11A11/65 

.25 

SK 

Schedule 

.167 

1/3 ->20/1 47 

17A17/65 

.05 

Qt 

_  Quality 

.167 

1/4A15/147 

4->4/65 

.30 

e2 

Effectiveness 

.167 

&  Efficiency 

1/5  A 12/1 47 

1 3->1 3/65 

.15 

Ip 

Intellectual 

.167 

1/6  A 10/147 

11A11/65 

.15 

Properties 

The  equal  weighting  method  simply  assigns  equivalent  weights  to  all  of  the 
variables.  The  rank  reciprocal  method  has  four  steps.  First,  each  variable  is  ranked  in 
order  of  relative  importance  (1  -  6  in  our  example).  Next  the  reciprocal  of  the  ranks  is 
taken  (1/1,  1/2,  1/3,  etc...).  The  resulting  fractions  are  then  added  together  using  a 
common  denominator  to  create  a  new  base  (60/60  +  30/60  +  20/60  +  15/60  +  12/60  + 
10/60  =  147/60).  Finally,  the  original  reciprocals  for  each  variable  are  divided  by  the 
new  base  (147/60)  with  the  resulting  distribution  being  used  for  weighting  (see  Table  2). 
The  equal  weighting  and  rank  reciprocal  methods  generally  do  not  provide  a  high  enough 
level  of  subjective  scrutiny  to  be  of  value  in  a  detailed  project.  With  the  pair-wise 
comparison  method,  the  decision  makers  are  provided  a  specific  number  of  points  to  be 
distributed  as  they  see  fit  between  the  variables  (65  points  for  the  example  in  Table  2). 
After  discussion,  each  variable  is  assigned  a  numerical  value.  The  sum  of  the  values  is 
then  used  as  the  denominator  for  the  variable  weighting,  with  the  numerator  being  the 
assigned  numerical  value.  Like  the  previous  two  methods;  pair-wise  also  fails  to  provide 
enough  ability  to  fine-tune  the  weighting  distribution  for  a  detailed  project.  The  direct 
assessment  method  uses  deductive  reasoning  to  detennine  and  assign  weights  to  each 
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variable.  Although  this  method  is  purely  subjective,  it  is  less  random  and  can  easily  be 
modified  as  necessary.  The  subjective  nature  of  direct  assessment  can  be  alleviated  to 
some  extent  by  using  a  number  of  technical  experts  to  develop  the  weighting  values  to  be 
used  in  the  formula. 

Once  weightings  have  been  assigned,  the  non-monetized  value  of  the  numerator 
variables  can  be  determined.  After  adding  the  variables  together,  the  resulting  numerator 
value  is  divided  by  the  asset  base  to  provide  an  NROI  output.  The  validity  of  this  NROI 
on  its  own  is  minimal.  Trend  analysis  is  required,  using  subsequent  alterations  to  the 
numerator  variables  for  the  first  ship  in  the  class  as  it  is  completed  and  compared  with 
independent  data  from  the  second  ship  in  the  class  as  it  progresses.  Essentially,  the  first 
NROI  developed  sets  a  baseline  that  is  used  to  compare  with  subsequent  outputs  for  the 
same  ship.  The  trend  data  from  the  first  ship  can  then  be  analyzed  to  determine  if 
priorities  can  be  adjusted  for  the  second  ship  in  order  to  improve  the  output. 
Additionally,  comparisons  can  be  made  between  ship  classes  provided  the  category 
modifiers  are  the  same  and  the  scope  of  the  project  is  similar;  as  is  the  case  with  the  RCS. 

3.  Monetized  Notional  ROI 

While  it  is  important  to  provide  weights  to  the  categories  used  in  the  numerator  of 
the  equation,  it  is  even  more  important  if  at  all  possible  to  use  monetary  values  for  the 
NROI  formula.  In  The  Bottomline  on  ROI,  while  discussing  the  detennination  of  ROI  in 
the  public  sector,  Patricia  Phillips  states  that  “converting  data  to  monetary  benefits  is 
critical... the  process  is  challenging,  particularly  with  soft  data,  but  can  be  methodically 
accomplished”  (2002:  73).  For  the  fonnula  to  be  most  applicable  for  the  purpose  of 
comparing  the  past,  present,  and  future  NROI  of  a  number  of  projects,  it  was  determined 
that  the  effort  must  be  expended  to  convert  the  data  to  monetary  values.  This  would  also 
serve  to  provide  an  NROI  as  near  to  traditional  as  possible  for  a  public  sector 
organization. 

As  formula  development  progressed  it  became  clear  that  the  final  formula  would 
be  a  combination  of  both  monetized  and  non-monetized  approaches.  The  cost  data  would 
be  actual  monetary  values  broken  down  into  a  number  of  categories.  Each  category 
would  then  be  modified  by  a  non-monetized  value  added  factor  (VAF)  determined  by  a 
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number  of  system  experts  using  the  direct  assessment  approach.  The  resulting  formula 
would  incorporate  the  essential  cost  elements  while  also  attempting  to  capture  the 
intangible  benefits  provided  by  the  public  sector  entity  in  the  absence  of  profit. 
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III.  PREVIOUS  EFFORTS  AT  DEVELOPING  ROI  IN  THE 

PUBLIC  SECTOR 


A.  AUSTRALIA 

1.  Scope  and  Reasoning  behind  the  Effort 

In  an  attempt  to  justify  government  acquisitions,  Australia’s  government 
continues  to  focus  on  improving  its  own  ability  to  develop  and  implement  ROI  criteria. 
In  a  society  vigorously  competing  for  scarce  public  resources,  receiving  the  best  value  for 
money  spent  has  become  central  to  government  policy.  The  Australian  Commonwealth 
demands  this  accountability.  In  response,  Australia’s  government  has  taken  strides  to 
emphasize  the  development  of  ROI  criteria  to  make  acquisition  decisions  throughout  its 
governmental  departments.  However,  we  were  unable  to  find  a  specific  example  where 
ROI  criteria  were  successfully  developed  and  implemented  by  a  governmental 
organization.  Dr.  Allen  Hawke,  Australia’s  previous  Secretary  of  Defense,  acknowledges 
the  public’s  frustration  with  their  lack  of  success  to  date.  According  to  Dr  Hawke’s 
address  in  February  of  2000,  “there  is  a  widespread  dissatisfaction  with  Defence’s 
Performance  (regarding  Australia’s  Defense  Organization  use  of  funds)... In  essence  we 
have  a  credibility  problem.” 

Australia’s  Department  of  Finance  and  Administration  (ADOFA)  is  responsible 
for  providing  direction  to  Australia’s  ministries  in  making  procurement  decisions. 
Instead  of  simply  choosing  the  lowest  cost  alternative,  ADOFA  emphasizes  the 
“achievement  of  value  for  money”  (2003).  Among  other  things,  this  method  weighs  the 
ability  of  the  alternatives  to  meet  the  stated  objectives,  the  reliability  and  reputation  of  the 
contractor,  and  the  whole  of  life  costs  instead  of  just  the  initial  procurement  cost.  Instead 
of  providing  structured  guidance  to  detennine  ROI,  ADOFA  provides  a  substantial  list  of 
things  to  consider  and  leaves  it  to  the  particular  agency  to  identify  and  weigh  those  things 
that  apply.  Due  to  the  unique  benefits  of  each  procurement  decision,  this  general 
approach  may  be  appropriate.  However,  recent  comments  from  Australia’s  Defense 
Procurement  Review  indicate  a  lack  of  success  thus  far  within  the  acquisition 
community.  The  review  concludes  with  the  following  comment; 
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Our  review  of  the  acquisition  process  has  led  us  to  conclude  that  there  is 
no  single  cause  of  the  failures  that  have  become  apparent  in  the 
development  of  capability  and  the  acquisition  and  support  of  defence 
equipment  (Australia  Dept,  of  Defence,  2003:  47). 

2.  Lessons  Learned 

The  Australian  government  does  acknowledge  the  need  to  consider  ROI  when 
making  procurement  decisions.  However,  by  merely  emphasizing  value  for  money  in 
broad  terms,  they  are  not  actually  implementing  quantitative  ROI  criteria  within  their 
government.  They  do  highlight  the  need  to  consider  many  important  factors  other  than 
costs  for  procurement  decisions  such  as  quality  and  contractor  performance.  Yet,  they  do 
not  provide  a  universal  method  for  considering  the  weighting  of  these  factors  so  that 
decisions  can  be  consistently  made  the  same  way  across  the  different  ministries.  Perhaps 
this  inconsistency  is  one  of  the  reasons  for  their  continued  lack  of  success  within  the 
Department  of  Defence  acquisition  community. 


B.  NEW  ZEALAND 

1.  Scope  and  Reasoning  behind  the  Effort 

The  following  ROI  case  concerning  the  Royal  New  Zealand  Navy  (RNZN)  is 
summarized  from  a  case  study  authored  by  Beryl  Ann  Oldham,  Paul  Toulson,  Brenda 
Sayers,  and  Graham  Hart  (2002).  The  RNZN  had  encountered  considerable  difficulty 
retaining  their  marine  engineers  (ME)  in  the  mid  1990s  due  to  high  attrition  rates.  The 
ME  community  is  responsible  for  many  of  the  complex  systems  aboard  the  RNZN’s  fleet 
ships  including  operation  and  maintenance  of  diesel  engines,  gas  turbines,  electrical 
generators,  and  air  conditioning  and  refrigeration  plants  (Royal  New  Zealand  Navy, 
2004).  The  attrition  problem  was  so  significant  that  the  ability  of  the  RNZN  to  maintain 
an  acceptable  operations  tempo  was  threatened.  Several  suggestions  were  made  in  an 
effort  to  reduce  the  ME  attrition  rate.  These  measures  included  improved  ME  career 
management  initiatives,  better  management  of  leave  and  maintenance  periods,  improved 
pay,  compensation  time  for  working  weekends,  and  the  more  controversial  Marine 
Engineer  Retention  Bonus  Scheme  (MERBS).  It  was  believed  that  implementation  of  an 
immediate  retention  bonus  was  imperative  to  control  the  attrition  problem  in  the  short  run 
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since  the  other  proposed  initiatives  would  be  slower  to  take  effect  (Oldham,  et  ah,  2002: 
132). 

The  MERBS  was  an  expensive  human  resource  endeavor  for  the  RNZN.  MERBS 
costs  included  both  administrative  program  set  up  costs  and  the  retention  payments  to 
personnel  themselves.  These  overall  costs  were  estimated  at  almost  five  million 
Australian  dollars  (144).  However,  the  MERBS  was  considered  a  successful  initiative 
since  it  did  reduce  attrition  rate  for  the  MEs  to  an  acceptable  level.  Unfortunately,  it  was 
difficult  to  detennine  just  how  successful  the  MERBS  was,  especially  considering  that 
other  retention  initiatives  were  occurring  simultaneously.  Consequently,  an  ROI  study 
was  conducted  to  determine  the  isolated  effect  of  the  MERBS  on  ME  retention  in  the 
RNZN. 

The  monetary  benefits  of  any  retention  program  are  the  avoided  expenses  for 
replacement  and  training  of  new  personnel  and  the  separation  costs  incurred  for  personnel 
leaving  the  military.  There  are  also  some  less  tangible  benefits  including  higher 
experience  levels,  improved  morale  and  increased  flexibility.  However,  in  order  to 
remain  objective,  the  focus  of  the  study  was  placed  on  the  monetary  benefits  achieved  by 
the  MERBS.  There  were  two  approaches  taken  to  isolate  the  monetary  benefits  of  the 
MERBS  on  retention. 

The  first  approach  was  more  subjective  in  nature  and  involved  the  use  of  a 
questionnaire  taken  by  both  the  participants  in  the  MERBS  and  their  managers.  The 
questions  were  tailored  to  evaluate  the  effectiveness  of  the  MERBS  and  its  isolated 
impact  on  the  ME  participants  to  stay  in  the  RNZN.  It  was  ultimately  detennined,  based 
on  these  questionnaires,  that  forty  one  percent  of  the  participants’  decisions  to  stay  in  the 
RNZN  were  influenced  by  the  MERBS.  These  forty  one  percent  were  then  asked  to  rate 
the  accuracy  of  their  answer  regarding  the  influence  of  the  bonus  payments  on  their 
decision  to  stay.  The  reply  indicated  they  were  ninety-three  percent  confident  in  the 
accuracy  of  their  answer  regarding  the  MERBS  influence  on  their  decision  to  stay  (138). 
The  actual  monetary  benefit  was  determined  by  multiplying  the  participants’  impact 
estimation  of  the  retention  payments  by  the  estimated  savings  of  retention  for  each  of  the 
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170  personnel  participating  in  the  MERBS.  The  estimated  savings  per  participant  and  the 
detailed  calculation  of  the  ROI  from  this  approach  is  shown  in  Table  3. 


Table  3.  Determining  ROI  with  the  Participant  Impact  Estimation  (Oldham,  et 

al.,  2002:  140, 145) 


Number  of  participants  in  the  MERBS  at  the  end  of  three  year  period  =170 _ 

Estimated  separation  cost  per  ME  leaving  the  service  =  $4,260 _ 

Average  replacement  cost  per  ME  leaving  the  service  =  $105,133 _ 

Percentage  of  decision  to  stay  influenced  by  retention  payments  =  41% _ 

Confidence  in  decision  to  stay  influenced  by  retention  payments  percentage  =  93% 

Participants  estimation  of  retention  payment’s  impact  =  0.41  x  0.93  =  0.38 _ 

Monetary  Benefits  =  ($4,260  +  $105,133)  x  170  x  0.38  =  $7,066,789 _ 

ROI  from  participant  impact  estimation  =  Net  Benefits/Program  Costs  x  100 

=  ($7,066,789  -  $4,926,504)/($4,926,504)  x  100 
=  43  % 


The  second  approach  involved  a  more  objective  approach  using  retention  trend 
data.  Predicted  turnover  of  ME  personnel  without  the  retention  payments  was  estimated 
based  on  historical  trends  in  both  ME  and  non-ME  personnel  prior  to  the  MERBS  period. 
Based  on  previous  trends  of  non-ME  and  ME  personnel  before  the  MERBS  initiative, 
ME  turnover  averaged  5.5  percent  higher  than  non-ME  personnel  (Oldham,  et  al.,  2002: 
141).  These  data  were  used  to  determine  an  expected  ME  turnover  without  the  retention 
payments.  This  was  compared  to  the  actual  turnover  during  the  MERBS  period  to 
detennine  an  actual  number  of  ME  participants  that  were  retained  as  a  result  of  the  bonus 
payments.  Based  on  this  comparison,  it  was  concluded  that  seventy-three  additional 
personnel  were  retained  during  the  MERBS  period  than  was  predicted  based  on  trend 
data  without  the  MERBS  initiative.  The  detailed  determination  of  the  ROI  based  on  this 
second  approach  is  shown  in  Table  4. 


16 


Table  4.  Determining  ROI  with  the  Forecasting  Method  (Oldham,  et  al.,  2002: 

144-45) 


Number  of  personnel  retained  attributed  to  MERBS  initiative  =  73 

Estimated  separation  cost  per  ME  leaving  the  service  =  $4,260 _ 

Average  replacement  cost  per  ME  leaving  the  service  =  $105,133 _ 

Monetary  Benefits  =  ($105,133  +  $4,260)  x  73  personnel  =  $7,985,689 

ROI  from  Forecasting  Method  =  Net  Benefits/Program  Costs  x  100 

=  ($7,985,689  -  $4,926,504)/  ($4,926,504)  x  100 
=  62  % 


2.  Lessons  Learned 

It  is  interesting  to  note  the  similarities  and  differences  between  the  two 
approaches.  Both  approaches  detennine  program  costs  the  same  way.  Even  though  the 
methods  of  detennining  monetary  benefits  are  very  different,  the  results  are  surprisingly 
similar.  The  participant  impact  estimation  yields  a  monetary  benefit  of  approximately 
seven  million  dollars  while  the  forecasting  method  yields  a  monetary  benefit  of 
approximately  eight  million  dollars.  However,  these  numbers  do  yield  significantly 
different  ROI  for  the  MERBS  initiative. 

The  question  then  becomes  which  approach  is  more  valid?  Both  approaches  are 
logical  and  defendable.  The  forecasting  approach  is  more  objective  since  it  is  based 
entirely  on  data  and  trend  analysis.  However,  the  shortcoming  is  that  it  does  not  entirely 
isolate  the  effect  of  the  MERBS  initiative  from  the  other  retention  initiatives  occurring 
simultaneously.  It  is  plausible  that  most  of  the  seventy-three  additional  personnel 
retained  were  a  result  of  the  MERBS,  but  it  is  certainly  possible  that  other  proposed 
initiatives  played  a  factor.  The  participant  impact  estimation  approach  is  clearly  more 
subjective  since  it  is  based  on  responses  from  a  questionnaire.  Conversely,  it  does  better 
address  the  isolated  effect  of  the  MERBS  initiative  through  the  inclusion  of  specific 
questions  in  the  questionnaire.  Unfortunately,  ROI  methodologies  based  on  non- 
traditional  methods  cannot  always  be  purely  objective.  This  is  what  makes  it  so 
challenging  in  the  public  sector.  The  objective  is  to  develop  a  credible  methodology, 
which  is  what  was  accomplished  here.  Therefore,  both  methodologies  are  valid  as  long 
as  their  shortcomings  are  kept  in  mind.  This  is  true  of  any  mathematical  formula. 
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A  lesson  learned  discussed  in  the  case  relates  to  data  collection.  Ideally,  the 
decision  to  perform  an  ROI  determination  is  made  prior  to  program  implementation. 
This  way  data  required  can  be  determined  and  recorded  while  the  program  is  taking 
place.  For  this  case,  the  decision  to  determine  the  ROI  for  the  MERBS  was  not  made 
until  after  the  program  was  implemented  and  well  under  way.  Therefore,  many  data 
collection  opportunities  were  missed.  The  recommendation  of  the  study  is  to  consider 
ROI  evaluations  and  associated  data  collection  needs  during  the  development  phase  of 
program  initiatives  if  possible  (Oldham,  et  ah,  2002:  146). 

Another  lesson  learned  from  the  case  was  the  extent  taken  to  keep  the  ROI 
methodology  simple.  They  could  have  attempted  to  determine  monetary  values  for  more 
complex  but  less  tangible  benefits.  For  example,  they  could  have  attempted  to  detennine 
the  monetary  benefit  of  the  increased  operations  tempo  or  increased  experience  levels 
available  because  of  the  higher  retention  rate.  However,  it  is  easy  to  imagine  that  this 
would  involve  potentially  long  and  complex  mathematical  formulae  and  additional 
subjectivity.  By  avoiding  these  attempts,  the  ROI  methodology  is  easier  to  understand 
and  more  credible. 

C.  UNITED  STATES  POSTAL  SERVICE 

1.  Scope  and  Reasoning  behind  the  Effort 

The  USPS  used  an  Economic  Value  Added  (EVA)  program  to  determine  their 
ROI  from  1996-2002.  EVA  was  calculated  by  detennining  the  net  operating  income  and 
subtracting  a  fee  proportional  to  the  cost  of  the  assets  used  to  produce  that  income 
(USPS,  1997).  This  difference  represented  a  positive  net  cash  flow  that  added  financial 
value  to  the  post  office.  A  higher  EVA  indicated  a  more  efficient  use  of  assets. 
Consequently,  senior  post  office  executives  were  rewarded  for  perfonnance  at  the  USPS 
based  on  this  figure.  This  provided  financial  incentive  for  post  office  employees  to  seek 
out  new  and  better  ways  to  improve  efficiency  within  the  organization.  This  program 
was  credited  with  contributing  to  the  $3.5  billion  in  net  income  earned  by  the  USPS  from 
1996-2000  (USPS,  2002).  However,  amid  strong  controversy  relating  to  the  calculation 
of  EVA  program  incentive  bonuses,  the  effort  was  abandoned  in  2002. 
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The  USPS  2003  Annual  Report  cites  the  continued  use  of  ROI  criteria  for  capital 
venture  decision-making.  Unlike  many  public  sector  organizations,  the  USPS  is  one  of 
the  few  public  sector  companies  that  generate  revenue.  This  facilitates  using  the 
traditional  method  of  calculating  ROI.  The  USPS  continues  to  invest  in  automation 
equipment  to  reduce  personnel  work  hours  in  mail  processing  and  delivery  (USPS,  2003: 
25).  The  cost  savings  realized  from  automation  is  then  compared  to  the  cost  of  acquiring 
the  required  equipment  for  making  procurement  decisions.  The  USPS  also  uses  a  Cash 
Flow/Capital  Expenditure  (CAPEX)  ratio  as  a  benchmark  for  assisting  in  making  capital 
purchase  decisions  (USPS,  2003:  24).  The  additional  yearly  cash  flow  to  operations  is 
compared  to  the  yearly  cash  outlays  to  support  the  project.  This  helps  to  determine  the 
attractiveness  of  the  proposed  project  and  the  required  need  to  borrow  funds  to  support  it. 

2.  Lessons  Learned 

The  basis  for  determining  economic  value  added  at  the  USPS  had  significant 
flaws.  The  idea  of  subtracting  additional  costs  from  increased  benefits  to  determine  value 
added  from  a  given  investment  is  sound.  However,  the  application  of  the  EVA  Variable 
Pay  Program  was  inconsistent  when  overall  USPS  performance  is  considered.  For 
example,  the  USPS  lost  $199  million  in  fiscal  year  2000  but  still  paid  out  over  $280 
million  in  performance  bonuses  (Lexington  Institute,  2001).  It  is  counterintuitive  for  an 
organization  to  lose  money  in  a  given  fiscal  year  and  still  pay  out  such  a  significant  sum 
of  money  in  perfonnance  bonuses.  In  fact,  the  reason  the  USPS  lost  money  that  year  was 
due  to  the  bonus  payouts.  Inconsistent  ROI  measurements  such  as  EVA  are  not  credible 
if  they  indicate  positive  results  when  other  metrics  such  as  negative  net  income  indicate 
to  the  contrary. 

The  USPS  uses  simple  and  intuitive  methods  for  detennining  ROI  for  evaluating 
investment  decisions.  The  use  of  cost  savings  as  a  basis  for  determining  ROI  is  a 
commonly  used  method  for  public  sector  organizations.  This  is  because  many 
procurement  decisions  made  by  public  sector  organizations  involve  investments  that  will 
ultimately  improve  efficiency.  If  these  efficiencies  are  able  to  be  quantified,  they  can  be 
used  as  a  basis  for  comparison  to  the  required  capital  expenditure  to  determine  an  ROI. 
The  CAPEX  ratio  used  by  the  USPS  to  evaluate  capital  purchase  decisions  is  also  an 

intuitive  way  to  determine  an  ROI.  Comparing  cash  flows  to  required  capital 
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expenditures  is  very  similar  to  a  net  present  value  calculation  commonly  used  in  the 
private  sector  to  evaluate  investment  alternatives.  It  is  important  to  emphasize  that  the 
USPS  does  generate  annual  revenues  which  lends  itself  to  the  use  of  traditional  ROI 
metrics.  This  is  uncommon  among  most  other  public  sector  organizations. 


D.  NAVY  DENTAL  CORPS 

1.  Scope  and  Reasoning  behind  the  Effort 

In  response  to  the  CNO’s  call  for  better  decision  making  tools,  the  Navy  Dental 
Corps  (NDC)  has  developed  a  simple  metric  to  detennine  ROI  at  the  branch  clinic  level. 
Captain  York,  the  navy  representative  at  the  Tri-Service  Center  for  Oral  Health  Studies 
(TSCOHS),  spearheaded  the  effort  to  determine  a  practical  method  for  defining  NDC’s 
return  for  investment  dollars.  While  not  using  the  traditional  method  of  ROI  that 
compares  earnings  to  assets,  this  effort  provides  an  easily  understood  metric  to  quantify 
performance  at  the  branch  clinic  level. 

NDC’s  ROI  formula  compares  a  branch  clinic’s  quarterly  output,  defined  as 
Dental  Weighted  Values  (DWVs),  to  its  required  investment  in  funding  dollars  and 
military  labor.  The  formula  is  as  follows  (Mitton,  2004): 


ROI  ( 0  )  -  \00)-{{BranchAPF  +  AnnualBranchMilab)  x  0.25) 

{Branch APF  +  Annual BranchMilab)  x  0.25 

DWV  ($)  -  Dental  Weighted  Values 

Branch  APF  ($)  -  Branch  Clinic  Annual  Planned  Funding 

Annual  Branch  Milab  ($)  -  Branch  Clinic  Military  Labor 


Branch  APF  is  the  operation  and  maintenance  funding  allocated  to  the  clinic.  Both 
Branch  APF  and  Annual  Branch  Milab  are  converted  to  quarterly  values  to  determine 
quarterly  ROI.  DWVs  and  Annual  Branch  Milab  are  determined  through  separate  data 
collecting  programs  described  next. 
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DWVs  are  determined  by  input  from  the  branch  clinic  into  a  program  known  as 
DENCAS.  The  clinic  enters  the  different  procedures  performed  on  a  given  day  using 
American  Dental  Association  (ADA)  procedural  codes,  known  as  Common  Dental 
Terminology  (CDT)  codes.  CDT  codes  are  converted  into  DWVs  that  are  essentially 
equal  to  one  hundred  dollars  worth  of  dental  services.  This  result  is  multiplied  by  one 
hundred  to  convert  the  DWVs  directly  into  dollars  for  use  in  the  ROI  formula. 

Annual  branch  clinic  labor  is  determined  by  the  collection  of  data  into  the 
Medical  Expense  and  Performance  Reporting  System  (MEPRS).  Branch  clinic 
employees  specifically  document  their  hours  worked  perfonning  a  variety  of  individual 
tasks  on  MEPRS  sheets.  Different  tasks  such  as  various  medical  duties,  training,  and 
even  leave/liberty  times  are  documented.  These  data  are  correlated  at  the  comptroller 
level  to  determine  military  labor  hours  and  is  converted  into  a  dollar  figure  based  on  the 
rank  and  rate  of  the  military  employees  working  at  the  clinic. 

2.  Lessons  Learned 

The  Navy  Dental  ROI  formula  discussed  above  is  no  longer  used  for  three 
reasons.  First,  there  is  significant  skepticism  regarding  the  quality  of  the  data  being 
tracked  for  use  in  the  calculation  of  ROI.  Specifically  mentioned  was  the  inaccurate  data 
collected  by  MEPRS.  Many  dental  employees  failed  to  log  their  hours  on  a  daily  basis; 
instead,  they  would  record  their  hours  on  a  weekly  or  monthly  basis.  This  brings  the 
accuracy  of  the  type  and  number  of  hours  into  question  due  to  the  delay  time  in 
recording.  Many  times  employees  would  wait  until  the  end  of  the  month  and  simply  log 
eight  hours  of  work  arbitrarily  for  each  day.  Commander  Mitton,  from  the  Navy  Bureau 
of  Medicine  and  Surgery  in  Washington  DC,  referred  to  this  popular  method  as  “logging 
straight  eights”  (2004). 

It  is  also  difficult  to  use  this  formula  for  comparison.  Different  branch  dental 
clinics  may  be  responsible  for  different  operating  costs.  For  example,  some  of  the  clinics 
are  responsible  for  paying  their  rent  and  utilities  while  other  clinics  are  provided  with 
these  resources  free  of  charge  directly  by  the  base  command.  This  directly  affects  the 
amount  of  Branch  APF  the  clinic  would  receive  and,  consequently,  affected  the  results  of 
the  ROI  formula. 
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Finally,  the  Navy  Dental  ROI  formula  does  not  include  many  of  the  cost  elements 
required  to  staff  and  operate  a  branch  dental  facility.  For  example,  large  expenses  such 
as  the  cost  of  training  Navy  dentists  and  dental  technicians  are  not  included.  Other  large 
costs  such  as  accession  bonuses  for  dentists  and  depreciation  expenses  for  major 
equipment  are  also  not  included.  Therefore,  ROI  for  the  Navy  branch  dental  clinics 
needed  to  be  more  adequately  defined. 

As  a  consequence,  Captain  York  developed  a  more  robust  formula  to  be  used  in 
calculating  ROI  for  Navy  Dental  clinics.  Although  similar  to  the  previously  discussed 
formula,  it  also  includes  many  of  the  lacking  cost  elements.  See  Table  5  on  the  following 
page  for  a  sample  calculation  using  Pearl  Harbor  for  FY  2000.  ROI  is  calculated  as 
shown  below  (York,  2004): 

....  ProductionValue-Cost  of  Production  ,  ^ 

ROI  (%)  = - - - x  1 00 

Total  Cost  of  Production 

Production  Value  is  detennined  similar  to  the  DWVs  calculated  in  the  original 
formula.  The  Cost  of  Production,  however,  includes  significantly  more  cost  elements. 
For  example,  note  the  system  costs  listed  in  Table  5.  These  are  the  allocated  training 
costs  to  the  Pearl  Harbor  dental  clinic  from  the  dental  training  pipeline.  These  were  not 
included  in  the  original  formula  but  are  real  costs  burdened  by  the  NDC  and  should  be 
included.  These  system  costs  are  divided  between  all  the  clinics  proportionately  based  on 
the  number  of  dental  technicians  and  dental  officers  employed  at  the  clinic. 
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Table  5 


Raw  Data:  Pearl  Harbor  Branch  Dental  Clinic  FY2000  (York,  2004) 


FY  00  COSTS  (EXCLUDING  DENTAL  COST  AND  PRODUCTION  ABOARD  SHIPS) 

1= 


Command/Branch: 


PEARL  HARBOR 


Number  of  command/BDC  Dental  Officers 
Number  of  military  dental  techs,  (incl.  ships) 


28 


2807 


(to  determine  system  costs) 


NDCs  NHs 

SYSTEM 

TOTAL 

Military  Pay  (Claimancy  18) 

$ 

5,708,382 

$ 

86,309 

$ 

5,794,691 

NDRI  Military  Pay 

$ 

18,694 

$ 

18,694 

Dental  Bonus  (above  composite) 

$ 

368,186 

$ 

368,186 

Dental  Accession  Bonus 

$ 

9,146 

$ 

9,146 

Civilian  (GS) 

$ 

449,140 

$ 

449,140 

Contracts 

$ 

548,086 

$ 

548,086 

Reserve  (composite  including  travel) 

$ 

529 

$ 

529 

Dental  Scholarships 

$ 

207,520 

$ 

207,520 

Outservice  Training  (DO) 

$ 

8,663 

$ 

8,663 

Officer  Indoctrination  School 

$ 

22,041 

$ 

22,041 

DT  "A"  School  (w/o  student  pay) 

$ 

68,517 

$ 

68,517 

DT  "A"  School  Student  Pay 

$ 

66,237 

$ 

66,237 

DT  "A  and  C"  School  Staff  Pay 

$ 

20,562 

DT  "C"  School 

$ 

57,157 

$ 

57,157 

RDH  Training  Program 

$ 

3,080 

$ 

3,080 

Officer  Recruiting 

$ 

5,828 

$ 

5,828 

Enlisted  Recruiting 

$ 

104,807 

$ 

104,807 

Recruit  Training 

$ 

103,770 

$ 

103,770 

MED-06/OODC 

$ 

1,359 

$ 

1,359 

NDRI  Operating  Funds 

$ 

40,757 

$ 

40,757 

Material  (equip  &  supplies) 

$ 

464,619 

$ 

464,619 

Equipment  Depreciation 

$ 

- 

Other  Procurements  Navy  (OPN) 

$ 

- 

Reimbursable 

$ 

48,457 

$ 

48,457 

Free  Receipts 

$ 

- 

All  Other 

$ 

385,049 

$ 

385,049 

Total  Cost  of  Production 

$ 

7,604,262  $ 

$ 

1,192,635 

1$ 

8,796,897  | 

Production  Value  * 

$ 

9,973,433  $ 

- 

$ 

9,973,433 

Production  Value  minus  Cost  of  Production 

ROI 

$ 

1,176,536 

13.4% 

*  Based  on  the  1995  ADA  Survey  of  Dental  Fees  (75th  percentile),  adjusted  to  1997  dollars  using 
the  Consumer  Price  Index  for  Dental  Services. 

Production  data  source:  Navy  Executive  Information  System  (EIS),  Dental  Weighted  Values. 


There  is  still  significant  variation  between  the  branch  dental  clinics  using  the 
improved  ROI  formula.  Table  6  compares  the  ROI  calculated  by  the  above  formula  for 
all  the  navy  branch  clinics.  Note  that  the  ROI  varies  from  approximately  negative  ten 
percent  to  over  one  hundred  percent.  These  variations  are  not  due  solely  to  differences  in 
performance  levels.  For  example,  NNDC  is  responsible  for  the  costs  of  the  Naval  Dental 
Postgraduate  School.  Due  to  this  fact,  the  NNDC  ROI  includes  the  impact  of  manpower 
costs  and  low  productivity  of  the  student-body  significantly  reducing  their  ROI. 
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Therefore,  even  though  this  fonnula  is  more  robust,  one  must  consider  more  than  just  the 
final  ROI  output  to  fairly  compare  commands. 


Table  6.  ROI  for  Navy  Branch  Dental  Clinics 


COMMAND 

COST 

PRODUCTION  VALUE 

PROD  -  COST 

ROI 

GREAT  LAKES 

$28,476,040 

$60,746,220 

$32,270,180 

113.3% 

OKINAWA 

$18,156,185 

$30,022,556 

$11,866,371 

65.4% 

MID  ATLANTIC 

$29,003,564 

$46,150,165 

$17,146,601 

59.1% 

PARRIS  ISLAND 

$10,810,353 

$15,747,470 

$4,937,117 

45.7% 

CAMP  PENDLETON 

$19,227,602 

$26,027,670 

$6,800,068 

35.4% 

SOUTHWEST 

$34,282,076 

$45,365,397 

$11,083,321 

32.3% 

SOUTHEAST 

$19,366,135 

$24,946,372 

$5,580,237 

28.8% 

GULF  COAST 

$14,541,505 

$18,741,273 

$4,199,768 

28.9% 

CAMP  LEJEUNE 

$17,474,322 

$22,031,239 

$4,556,917 

26.1% 

EUROPE 

$12,454,115 

$15,642,759 

$3,188,644 

25.6% 

PEARL  HARBOR 

$8,796,897 

$9,973,433 

$1,176,536 

13.4% 

FAR  EAST 

$13,896,531 

$15,376,018 

$1,479,487 

10.6% 

NORTHEAST 

$11,073,148 

$12,213,250 

$1,140,102 

10.3% 

NORTHWEST 

$9,987,892 

$9,711,583 

-$276,309 

-2.8% 

NNDC 

$32,151,242 

$29,096,131 

-$3,055,111 

-9.5% 

ALL  NDCs 

$279,697,607 

$381,791,536 

$102,093,929 

36.5% 
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IV.  NOTIONAL  ROI  FORMULA  DEVELOPMENT  FOR  SPAWAR 
PEO  C4I  AND  SPACE  NEW  SHIP  CONSTRUCTION 

A.  T-AKE  PROJECT 


Figure  1.  T-AKE  1:  USNS  Lewis  and  Clark  (Global  Security,  2004) 

1.  Ship  Class  Background  Data  and  Building  Plan 

The  T-AKE  class  of  ships  will  provide  Naval  forces  at  sea  with  replenishment 
capability  involving  ammunition,  food,  repair  parts,  consumables,  and  to  a  lesser  extent 
marine  and  aviation  fuel.  The  initial  plan  is  for  twelve  ships  to  be  constructed  in  three 
distinct  sets  of  four  ships  each  known  as  flights  (Program  Executive  Office,  2003:  iv). 
The  role  of  PEO  C4I  SC  with  regard  to  the  T-AKE  is  to  “design,  procure,  integrate,  test, 
deliver,  install,  train,  and  logistically  support  the  T-AKE  C4I  suite  of  equipment”  (iv). 

SPAWAR  Systems  Center  Charleston  (SSCC)  was  tasked  with  configuring  the 
RCS  portion  of  the  overall  installation.  The  RCS  consists  of  all  shipboard  equipment 
used  for  voice  and  data  communications.  In  addition  to  T-AKE,  SSCC  is  also 
responsible  for  the  RCS  configuration  on  the  LPD-17,  LHD-8,  and  CVN-68  classes  of 
ships.  By  retaining  sole  responsibility  for  the  RCS  across  a  number  of  platforms,  SSCC 
translates  lessons  learned  from  one  class  of  ships  directly  onto  other  classes,  limiting  the 
duplication  of  errors  prevalent  in  stove-piped  efforts.  The  assemblage  of  concentrated 
experience  and  talent  at  SSCC  provides  an  environment  whereby  problems  can  be 
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worked  through  quickly  and  efficiently  and  most  importantly  where  developed  better 
business  practices  can  significantly  improve  the  deliverables. 

Initial  engineering  efforts  to  plan  the  design  for  the  T-AKE  RCS  began  in  the 
fourth  quarter  of  2001.  This  effort  continued  into  early  2004  with  necessary 
modifications  made  along  the  way.  Concurrently,  design  of  the  Test  Integration  Facility 
(TIF)  and  procurement  of  equipment  began  in  the  fourth  quarter  of  2002  and  continued 
on  into  the  third  quarter  of  2004.  Transfer  of  the  tested  RCS  will  be  conducted  in  the  first 
quarter  of  2005.  USNS  Lewis  and  Clark,  as  T-AKE  1  will  be  called  when  completed,  is 
being  built  at  the  NASSCO  shipyard  in  San  Diego,  CA.  She  is  expected  to  be 
commissioned  in  mid  2005. 

2.  Turnkey  Approach 

With  the  conventional  shipbuilding  approach,  all  equipment  destined  for 
installation  on  the  ship  had  to  be  identified  well  in  advance,  typically  5-9  years  with 
regard  to  RCS  equipment  (Program  Executive  Office,  2003:  2).  This  extensive  lead  time 
led  to  many  problems  such  as  cost  overruns  due  to  technology  refresh  requirements, 
engineering  rework  caused  by  the  need  to  make  modifications  to  support  new  equipment, 
and  in  some  cases  no  cost  impact  but  rather  an  efficiency  impact  as  outdated  or  even 
obsolete  equipment  were  left  on  board.  A  more  novel  method  called  the  Turnkey 
Approach  is  now  being  utilized  by  SPAWAR. 

With  the  Turnkey  Approach,  design  deadlines  are  developed  with  regard  to  space 
and  power  requirements  but  responsibility  for  installation  rests  with  PEO  C4I  SC  and  not 
with  the  shipbuilder.  In  this  way  the  shipbuilder  can  continue  on  with  ship  construction, 
providing  the  spaces  and  cable  routes  as  prescribed  by  PEO  C4I  SC  but  leaving  the 
installation  and  integration  of  the  actual  equipment  up  to  PEO  C4I  SC.  This  allows  for  a 
dramatic  reduction  in  lead  time  for  the  procurement  of  the  equipment  from  5-9  years 
down  to  2  years  (2). 

The  key  element  to  successful  implementation  of  the  Turnkey  Approach  is  the 
TIF  located  at  SSCC.  The  TIF  is  essentially  a  mock  up  of  the  eventual  RCS  for  a 
particular  ship.  The  actual  equipment  that  will  be  used  on  the  ship  is  installed  and 
integrated  in  advance  at  the  TIF  vice  the  conventional  method  onboard  ship.  This  allows 
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for  the  technicians  to  work  out  the  problems  coincident  with  any  such  endeavor.  Once 
the  TIF  RCS  is  up  and  running  at  top  efficiency  the  components  are  packed  and  shipped 
to  the  shipboard  site  for  actual  installation.  In  stark  contrast  to  conventional  methods,  the 
installation  of  the  RCS  systems  is  not  normally  done  until  approximately  140  days  prior 
to  the  builder’s  ship  trials.  (Program  Executive  Office,  2003:  3) 

B.  NROI  FORMULA  SEQUENCE  OF  DEVELOPMENT 

The  sequence  of  development  for  the  SPAWAR/PEO  C4I  SC  NROI  application 
spanned  over  ten  months  of  time  and  involved  several  face-to-face  and  video 
teleconference  (VTC)  meetings  with  Command  personnel.  Since  SPAWAR  is  a  public 
sector  organization,  they  do  not  report  a  net  income  which  would  traditionally  be  used  in 
the  private  sector  to  determine  a  corporation’s  ROI.  Therefore,  our  task  was  to  develop 
an  NROI  formula  that  would  credibly  measure  the  PEO  C4I  SC  ROI  for  the  T-AKE 
project.  This  evolutionary  process  produced  four  iterations  before  deciding  on  the  final 
NROI  formula  in  an  effort  to  fairly  represent  the  added  value  contributed  by 
SPAWAR/PEO  C4I  SC  to  the  T-AKE  project. 

The  NROI  formula  development  process  had  to  be  approached  considering  a 
number  of  criteria  in  an  effort  to  make  it  both  acceptable  and  useful.  Nine  non-mutually 
exclusive  criteria  applied  in  development  of  the  NROI  method  included:  simplicity  and 
intuitiveness,  feasibility  of  implementation,  credibility,  theoretical  adequacy  and  utility, 
flexibility,  validity,  ability  to  be  replicated,  comprehensiveness,  and  consideration  of  all 
program  costs.  These  must  all  be  taken  into  account,  as  per  advice  provided  in  Patricia 
Phillips  The  Bottomline  on  ROI,  and  are  summarized  below  as  related  to  development  of 
the  NROI  for  PEO  C4I  SC  (2002:  60-62). 

Simplicity  and  Intuitiveness:  NROI  formulae  should  be  straightforward  and 
easily  understood.  Complex  NROI  formulae  made  up  of  long  mathematical  equations 
serve  little  purpose  and  can  be  unnecessarily  confusing  to  those  they  are  intended  to 
assist.  NROI  formulae  should  be  quickly  comprehended,  thus  providing  managers  a 
useful  tool  to  understand  the  information  provided  for  their  use. 
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Feasibility  of  Implementation:  Implementation  of  a  new  performance 

measurement  for  the  organization  can  be  time  consuming  and  costly.  For  example,  the 
requirement  of  a  new  data  collection  method  to  measure  performance  might  require 
significant  managerial  resources  and  dollars  to  employ.  Therefore,  consideration  must  be 
given  to  the  feasibility  of  the  elements  that  make  up  the  NROI  formula  in  order  to 
minimize  this  burden.  In  cases  where  it  is  possible,  utilization  of  existing  data  collection 
methods  (provided  they  are  relevant)  is  recommended. 

Credibility:  The  developed  NROI  formula  must  be  considered  credible  by  those 
whose  efforts  it  is  intended  to  measure  and  by  the  managers  reviewing  the  results.  A 
performance  measurement  not  considered  credible  by  those  tasked  to  implement  it  is 
useless.  It  will  either  not  be  implemented  as  it  was  intended  or  will  simply  be 
disregarded.  Whenever  possible,  it  is  imperative  to  use  objective  measures  to  represent 
the  outputs  that  are  used  to  measure  performance.  In  cases  where  this  is  not  possible, 
steps  must  be  taken  to  minimize  the  formula’s  subjectivity.  Finally,  a  conservative 
approach  must  be  considered  when  choosing  between  formula  alternatives  in  order  to 
preserve  the  credibility  of  the  process. 

Theoretical  Adequacy  and  Utility:  The  NROI  formula  developed  must  be 
consistent  with  other  measurements  of  ROI.  According  to  Patricia  Phillips;  “The  ROI 
methodology  must  be  theoretically  sound  and  based  on  generally  accepted  practices” 
(2002:  61).  The  numerator  must  represent  the  organization’s  net  benefits  and  the 
denominator  must  represent  the  total  costs  associated  with  achievement  of  those  net 
benefits.  This  enables  utility  similar  to  private  sector  uses  of  ROI. 

Flexibility:  The  developed  NROI  formula  must  be  flexible  enough  to  be  used  in 
several  applications.  This  includes  the  assessment  of  the  ROI  during  various  phases  of 
the  T-AKE  project  as  well  as  the  program’s  overall  ROI. 

Validity:  Validity  of  the  developed  NROI  formula  must  be  verified  through 
actual  application  to  the  organization.  It  should  produce  output  consistent  with 
expectations.  For  example,  an  increase  in  non-value  added  costs  should  reduce  the 
organization’s  NROI. 
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Replicable:  The  formula  should  allow  replication  of  results  by  independent 
sources  provided  the  same  data  are  available.  Once  developed  and  approved,  the  formula 
should  be  able  to  be  applied  objectively,  even  though  it  may  contain  some  subjective 
elements. 

Comprehensiveness:  All  relevant  components  of  an  organization  that  add  value 
to  the  final  products  and  services  should  be  represented  in  the  formula  if  possible.  Some 
outputs  may  be  difficult  to  quantify,  and  must  be  noted  as  intangibles.  However, 
maximum  inclusion  of  benefits  is  necessary  for  fair  valuation  of  the  organization’s 
outputs. 

Consideration  of  All  Program  Costs:  Fair  assessment  of  the  organization’s  ROI 
also  involves  accurately  accounting  for  all  the  program  costs  that  are  incurred  in 
producing  the  output.  These  include  the  direct  costs  of  the  project  as  well  as  a  fair  share 
of  the  corporate  costs  or  overhead  incurred  by  the  organization. 

The  most  basic  ROI  formula  is  shown  below  (Phillips,  2002:  19): 


ROI  (%)  =  Net  Program  Benefit  %  m 
Program  Costs 

This  was  chosen  as  the  baseline  template  to  be  used  in  the  development  of  the  NROI 
formula  for  PEO  C4I  SC.  Net  program  benefits  are  benefits  minus  the  costs  of  achieving 
those  benefits.  Program  costs  will  include  both  direct  costs  of  the  program  as  well  as  a 
fair  share  of  the  corporate  costs.  An  ROI  determined  directly  from  this  formula  yields  a 
percentage  ROI  similar  to  the  ROI  fonnulas  used  in  the  private  sector. 

1.  1st  Iteration 

Dr.  Lawrence  R.  Jones,  RADM  George  F.  A.  Wagner/SPAWAR/PEO  C4I  and 
Space  Professor  of  Public  Management  in  the  Graduate  School  of  Business  and  Public 
Policy  (finance  curriculum)  at  the  Naval  Postgraduate  School  (NPS)  in  Monterey 
spearheaded  a  new  approach  to  solving  the  public  sector  ROI  dilemma.  Beginning  in  late 
2003  Dr.  Jones  worked  closely  with  (RADM  ret.)  William  Marshall,  an  independent 
contractor  from  American  Systems  Corporation  (ASC)  serving  as  a  consultant  to  PEO 
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C4I  SC  in  San  Diego.  Their  goal  was  to  think  through  the  application  of  ROI  to  the  Navy 
and  PEO  C4I  SC  in  response  to  the  call  for  such  action  by  the  CNO.  A  value-added 
approach  to  determine  an  NROI  was  developed  by  Jones  with  assistance  from  Marshall, 
Tom  Sommers  and  other  PEO  C4I  SC  staff.  After  several  months  of  serious  dialogue  on 
the  task  at  hand,  the  decision  was  made  by  Joe  Mayer,  head  of  PEO  C4I  SC,  to  move 
forward  with  a  specific  project.  The  platfonn  to  be  modeled  was  the  C4I  suite  of  a  new 
class  of  supply  ship  (T-AKE).  The  role  of  PEO  C4I  SC  is  the  installation,  integration  and 
testing  of  the  suite  as  well  as  the  personnel  training  required  for  the  operators.  Ironically, 
the  first  ship  and  class  name  is  Lewis  and  Clark,  named  for  the  two  explorers  who  led  a 
visionary  project  of  exploration.  (United  States  Dept,  of  Defense,  2003) 

The  original  NROI  fonnula  was  developed  with  the  help  of  a  number  of 
SPAWAR/PEO  C4I  SC  representatives,  led  by  Jones,  Sommers  and  Marshall,  at  their 
semi-annual  meeting  in  December  2003  (Jones  et  ah): 


NROI  (%)  = 


(Cp  +  Ct  +  Sk  +  Qt  +  E2  + Ip) 

— - - - C2xioo 

Asset  Base 


The  numerator  inputs  will  be  weighted  according  to  the  relative  importance  to  one 
another  and  would  ultimately  be  multiplied  by  the  cost  of  labor  hours  devoted  to  that 
category.  As  described  previously,  the  weighting  is  very  important  and  was  to  be 
determined  using  primarily  subjective  analysis  by  the  key  players.  It  is  important  now  to 
take  a  closer  look  at  each  of  the  numerator  inputs  individually  with  the  realization  that  the 
formula  inputs  were  still  in  the  development  stage.  Each  of  the  sub  categories  would  also 
be  weighted  within  each  variable  using  one  of  the  four  aforementioned  methods. 

Capability  (Cp)  addresses  the  integration  and  testing  of  the  hardware  and 
software,  the  scalability  of  the  suite,  and  the  ease  with  which  the  technology  can  be 
updated.  Cost  (Ct)  accounts  for  the  time  versus  the  resources  used,  benefits  from 
consolidated  buys,  and  lessons  learned.  Schedule  (Sk)  assesses  the  TIF  integration 
process,  flexibility,  and  management  performance.  Quality  (Qt)  focuses  on  metrics, 
testing,  and  interoperability  standards.  Effectiveness  and  Efficiency  (E2)  determines  the 
effect  of  economies  of  scale,  lessons  learned,  and  standard  practices.  Lastly,  Intellectual 
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Properties  (Ip)  represent  the  expertise,  experience,  and  training  of  the  personnel  involved 
in  the  process  (Jones  et  ah,  2003). 

2.  2nd  Iteration 

A  second  iteration  of  this  formula  was  developed  at  the  May  2004  SPAWAR 
PEO  C4I  and  Space  semi-annual  meeting.  It  is  shown  below  including  the  minor 
modifications  that  were  made  to  ease  the  data  collection  for  the  formula: 


NROI  (%)  = 


(Cp+Sk  +  Qt+Ip)xl00 

Asset  Base 


The  Cost  and  Effectiveness  and  Efficiency  value  added  categories  were  dropped  from  the 
formula.  This  was  done  as  a  result  of  a  consensus  opinion  that  these  two  categories  were 
largely  indistinguishable  from  elements  already  present  in  the  remaining  four  categories. 
The  major  obstacle  still  before  the  effort  at  this  point  was  developing  a  data  collection 
method  that  would  allow  for  easy  translation  of  labor  hours  into  the  remaining  four 
categories. 

Unfortunately,  even  with  the  modifications  to  the  baseline  formula,  the  2nd 
iteration  was  later  determined  to  be  technically  unfeasible  for  two  primary  reasons.  First, 
the  value  added  categories  as  developed  remained  too  vague.  It  was  concluded  that  it 
would  be  too  difficult  to  objectively  separate  the  labor  hours  from  the  T-AKE  project  into 
the  individual  categories.  Also,  some  of  the  labor  hours  might  fit  into  more  than  one  of 
the  categories.  For  example,  an  hour  of  labor  spent  installing  equipment  may  contribute 
to  both  Capability  (Cp)  and  Quality  (Qt).  How  would  these  “double  value”  hours  be 
separated  to  best  represent  their  contribution?  Secondly,  data  collection  was  not  already 
in  place  to  support  this  formula.  Gathering  data  would  have  required  the  creation  of  new 
data  collection  methods  thought  to  be  too  cumbersome  and  costly.  It  was  also 
determined  that  it  would  not  be  feasible  to  develop  credible  values  for  average  labor  rates 
for  use  in  the  numerator  since  they  vary  considerably  throughout  the  organization. 
Consequently,  without  quality  data  available  to  support  the  1st  or  2nd  iterations,  a  change 
of  approach  was  in  order. 
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3.  3rd  Iteration 

The  3rd  iteration  of  the  baseline  formula  was  developed  at  a  conference  held  in 
San  Diego  at  ASC  on  July  23,  2004.  This  formula  represented  a  much  refined  approach 
to  the  problem  relying  on  a  much  smaller  core  group  of  key  players  as  opposed  to  the 
more  open  ended  discussions  at  the  semi-annual  meetings.  In  the  process  of  lengthy 
discussion  about  data  availability,  validity  and  applicability  with  Travis  Tillman,  the 
Platform  Manager  for  the  T-AKE  project,  the  third  version  of  the  NROI  formula  was 
developed  by  Jones,  Bigham  and  Goudreau  as  shown  below: 


PMxl.l+ENG  xl.4+SMxl.l5+PRxl.4+INSTxl.3+SSxl.2+MMxl.l+Avoidanc£osts-Reworl<Costs 

NROI(%)  =  ( - 1)jc100 

RadioControlSuiteProgramCosts 

WBS  CATEGORIES 

PM  -  Program  Management 
ENG  -  Engineering 
SM  -  Supportability  Management 
PR  -  Production  (Integration) 

INST  -  Installation 
SS  -  Shipyard  Support 
MM  -  Material  Management 

The  numerator  of  the  formula  was  changed  considerably  as  compared  with  the  first  two 
iterations.  There  are  four  significant  differences,  the  use  of  SSCC’s  existing  cost  tracking 
method  of  Work  Breakdown  Structure  (WBS)  categories,  the  utilization  of  VAFs,  the 
addition  of  avoidance  costs,  and  the  capturing  of  rework  costs. 

First,  rework  costs  incurred  due  to  PEO  C4I  SC  actions  would  be  separated  out  of 
the  value  added  categories.  Rework  does  not  add  value  to  the  project  and,  consequently, 
should  not  enhance  the  NROI.  Second,  based  on  a  recommendation  that  arose  during  a 
phone  conference  with  Dr.  Patricia  Phillips,  the  president  and  CEO  of  the  ROI  Institute,  a 
cost  avoidance  category  was  added  to  the  numerator  (2004).  The  purpose  of  this 
adjustment  is  to  capture  any  costs  avoided  as  a  consequence  of  the  role  PEO  C4I  SC 
plays  in  coordinating  the  expenditure  of  DOD  resources.  For  example,  if  PEO  C4I  SC 
makes  a  bulk  buy  of  radio  room  equipment  to  support  multiple  DOD  contracts  and  saves 

a  particular  sum  of  money,  this  would  be  considered  a  cost  avoided.  If  a  piece  of  this 
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equipment  is  then  used  in  the  T-AKE  ship  project,  some  of  this  avoided  cost  should  be 
credited  to  the  project  as  an  avoidance  cost  and  improve  the  project’s  NROI. 

Lastly,  the  WBS  categories  parallel  data  already  kept  by  the  T-AKE’s  platform 
manager  with  only  minor  modifications.  It  was  determined  that  the  same  process  of 
applying  weighting  factors  could  also  be  used  to  determine  VAFs  to  modify  the  seven 
existing  WBS  categories  in  order  to  represent  the  value  added  by  PEO  C4I  SC  to  the  RCS 
for  the  T-AKE  project.  For  example,  engineering’s  VAF  of  1.4  indicates  that  every 
dollar  spent  on  engineering  gives  the  final  product  a  total  value  of  $1.40.  A  detailed 
description  of  the  WBS  categories  and  VAFs  follows. 

4.  WBS  Categories  and  Associated  VAFs 

The  WBS  categories  used  in  the  numerator  of  the  NROI  formula  reflect  the  seven 
categories  commonly  used  across  all  SSCC  RCS  Programs.  The  numerical  values  for  the 
VAFs  for  each  of  the  WBS  categories  were  initially  proposed  by  Travis  Tillman  at  the 
July  ASC  conference.  However,  in  an  effort  to  strengthen  the  accuracy,  validity  and 
credibility  of  the  weighting  methodology,  other  experts  within  SSCC  were  asked  for  their 
valuation  of  the  different  WBS  cost  categories  via  email  correspondence  at  the  beginning 
of  August.  The  following  five  members,  with  their  associated  titles,  then  participated 
with  the  authors  in  a  video  teleconference  (VTC)  meeting  on  Aug  6,  2004  during  which 
final  agreement  on  the  VAFs  was  established: 

Mr.  Travis  Tillman  -  T-AKE  1  Class  Platform  Manager 

Mr.  Ken  Ayers  -  LPD-17  Radio  Communications  System  Program  Manager, 

Mr.  Jim  Burgess  -  CVN-68  Radio  Communications  System  Program  Manager 

Mr.  Mike  Cullison  -  LHD-8  Radio  Communications  System  Program  Manager 

Mr.  Kevin  Gerald  -  CVN-68  Radio  Communications  System  Project  Engineer 

The  VAFs  for  each  of  the  WBS  elements  were  developed  by  consensus  among  this  group 
of  SSCC  civil  servants  who  collectively  have  over  60  years  of  experience  designing, 
engineering,  installing,  integrating,  testing,  and  supporting  radio  communications  systems 
for  Ship’s  Construction,  Navy  (SCN)  platforms.  Experience  of  these  members  includes  a 
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number  of  different  delivery  approaches  across  several  platfonns  spanning  the  last  20 
years. 

The  following  discussion  of  the  different  WBS  categories  and  their  associated 
VAFs  is  derived  primarily  from  explanations  provided  by  Travis  Tillman  in  person,  via 
VTC  and  through  email  (2004).  Our  explanation  serves  as  a  summary  of  comments 
made  principally  during  the  VTC  where  the  VAF  values  were  decided  upon  by  the 
participating  group  members. 

Program  Management  (PM):  A  weighting  factor  of  1.2  was  applied  to  program 
management  based  on  the  agreed  amount  of  value  added  to  dollars  spent  within  this 
category.  RCS  managers  have  the  required  education  and  experience  to  make  decisions 
within  their  programs  that  are  based  on  best  engineering  practice  and  past  experience. 
The  RCS  managers  also  have  the  benefit  of  discussing  and  sharing  issues  with  each  other 
at  SSCC.  Currently,  the  four  SSCC  RCS  managers  have,  on  average,  15  years  of  direct 
SCN  integration  and  installation  experience.  SSCC  managers  are  also  using  a  number  of 
project  management  tools.  For  example,  the  LPD-17  Class  program  is  using  the 
Capability  Maturity  Model1*  Integration  (CMMI®)  project.  CM  Ml"  is  a  collaborative 
effort  to  provide  models  for  achieving  product  and  process  improvement.  The  primary 
focus  of  the  project  is  to  build  tools  to  support  improvement  of  processes  used  in  the 
development  and  sustenance  of  systems  and  products.  The  output  of  the  CMMI®  project 
is  a  suite  of  products,  which  provides  an  integrated  approach  across  the  enterprise  for 
improving  processes,  while  reducing  the  redundancy,  complexity,  and  cost  resulting  from 
the  use  of  separate  and  multiple  capability  maturity  models  (CMM®s). 

Engineering  (ENG):  A  weighting  factor  of  1.4  was  applied  to  engineering  based 
on  the  agreed  amount  of  value  added  to  dollars  spent  within  this  category.  The  weighting 
factor  of  1.4  was  applied  to  engineering  based  on  the  RCS  managers’  past  experience 
developing  engineering  products  for  a  number  of  SSCC  delivered  radio  control  systems 
for  platfonns  currently  in  construction  as  well  as  those  recently  delivered  to  the  fleet 
(Current:  CVN  70/77,  LHD  8,  LPD  17-21,  T-AKE  1  -  4;  Recent:  CVN  69/76).  The 
core  team  of  engineers  has  collectively  worked  on  over  15  RCS  programs  during  the  past 
20  years.  SSCC  uses  a  core  team  of  engineers,  leverages  engineering  products  from  one 
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hull  to  the  next,  uses  best  engineering  practices,  to  include  baseline  management,  and 
uses  an  extensive  number  of  Standard  Operating  Procedures  (SOP)  to  improve  efficiency. 

SSCC  has  also  developed  engineering  products  that  are  tailored  to  the  specific 
platform’s  requirements.  For  example,  once  a  specific  drawing  is  developed  for  a  given 
communication  system,  such  as  High  Frequency  (HF),  it  becomes  the  starting  point  for 
follow-on  platforms.  Drawings  can  be  sized  or  scaled  to  meet  follow-on  platform 
requirements.  The  ability  to  leverage  drawings  from  one  platfonn  to  the  next  is  one  way 
SSCC  achieves  a  significant  cost  savings  in  engineering.  SOPs  are  also  used  as  a  tool  for 
SSCC  to  minimize  cost.  SOPs  are  constantly  refined  to  improve  the  quality  of 
engineering  products  developed.  One  of  the  most  beneficial  practices  employed  by  SSCC 
is  Baseline  Management.  Baseline  Management  allows  multiple  agencies,  as  well  as 
multiple  contractors,  to  stay  on  the  same  page.  Although  engineering  products  and 
associated  development  are  leveraged  across  multiple  hulls  at  SSCC,  the  final  products 
are  still  tailored  to  the  specific  platform  and  allow  for  the  resultant  Infonnation 
Capabilities  Document,  cable  run  sheets,  and  “As  Built”  drawings  to  be  a  comprehensive 
set  of  documentation  to  be  used  by  both  ship’s  force  and  future  installers.  Because  these 
final  tailored  packages  are  designed  and  reviewed  before  and  during  the  Production  phase 
of  acquisition,  the  latest  available  technology  insertion  can  be  employed  to  provide  the 
fleet  with  the  most  recent  equipment  the  ship’s  C4I  baseline  will  allow. 

Material  Management  (MM):  A  weighting  factor  of  1.1  was  applied  to  material 
management  based  on  the  agreed  amount  of  value  added  to  dollars  spent  within  this 
category.  Although  most  of  the  efforts  associated  with  material  management  will  occur 
regardless  of  the  acquisition  strategy  or  approach,  the  SSCC  SCN  Material  Management 
team  has  many  years  of  experience  in  this  category  and  has  developed  and  consistently 
refined  efficient  standard  operating  procedures.  The  labor  for  the  team  and  lessons 
learned  are  shared  across  multiple  platforms  constantly  improving  work  practices. 

Supportability  Management  (SM):  A  weighting  factor  of  1.15  was  applied  to 
supportability  management  based  on  the  agreed  amount  of  value  added  to  dollars  spent 
within  this  category.  The  SSCC  has  a  core  team  of  experts  that  have  been  supporting  the 
SCN  process  locally  for  the  past  eight  years.  Most  of  the  efforts  associated  with 
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supportability  management  are  common  across  all  platforms  and  will  occur  regardless  of 
acquisition  strategy  or  approach.  However,  the  crew  familiarization  and  limited 
maintenance  training  are  developed  for  the  specific  platfonn.  The  development  of  the 
supporting  documents  and  classroom  packages  are  leveraged  from  previous  performed 
training.  This  efficient  process  provides  a  cost  savings  over  developing  these  training 
programs  from  the  ground  up  each  time.  SSCC  has  established  a  Supportability 
Integrated  Product  Team  (SIPT)  that  crosses  all  past,  current,  and  future  platfonns.  The 
SIPT’s  sharing  of  lessons  learned  across  multiple  platforms  is  also  incorporated  into  the 
weighting  factor. 

Production/Integration  (PR):  A  weighting  factor  of  1.4  was  applied  to 
production/integration  based  on  the  agreed  amount  of  value  added  to  dollars  spent  within 
this  category.  There  is  significant  “value  added”  in  the  integration  and  testing  that  SSCC 
performs  in  each  TIF.  The  production  efforts  SSCC  performs  serve  several  functions. 
They  allow  for  integration  and  testing  of  radio  room  components  to  occur  prior  to 
installation  on  the  ship.  SSCC  production  efforts  include  providing  the  cables, 
connectors,  back  shells  and  other  systems  not  sponsored  by  the  hardware  providers,  but 
that  are  required  to  make  individual  RF/switching  systems  into  an  integrated  RCS 
system.  The  test  and  integration  process  is  used  to  allow  individual  systems  to  be  pre¬ 
assembled  and  tested  in  a  lab  environment  vice  onboard  ship,  where  there  is  potential  for 
damage  to  occur  due  to  the  industrial  work  ongoing  during  new  ship  construction. 

The  TIF  provides  the  first  opportunity  to  test  systems  for  interoperability  with 
other  new  and  legacy  systems.  The  TIF  environment  also  allows  for  a  full  “mock  up”  of 
the  shipboard  spaces  that  is  used  to  arrange  equipment  in  an  effective  and  efficient 
environment  as  well  as  ensure  Human  Machine  Interfaces  (HMI)  are  located  in  the 
optimal  location  for  monitoring  the  equipment.  This  is  essential  to  ensure  that  actual 
onboard  operators  of  the  equipment  can  adequately  monitor  operations  within  the 
proposed  configuration  of  the  radio  room.  Additionally,  the  TIF  “mock  up”  ensures  the 
required  access  space  for  maintainers  and  operators  is  maintained. 

Installation  (INST):  A  weighting  factor  of  1 .3  was  applied  to  installation  based 
on  the  agreed  amount  of  value  added  to  dollars  spent  within  this  category.  This  was 
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derived  from  SSCC’s  vast  experience  and  lessons  learned  from  previous  SCN  RCS 
installations.  Many  of  the  efforts  associated  with  the  installation  process  are  common 
across  different  acquisition  strategies  and  approaches  (ie.  government  vs.  shipbuilder 
installation).  However,  the  experience  of  SSCC  personnel  and  associated  contractors 
greatly  decrease  the  installation  and  test  time  needed  during  the  construction  window. 
Also,  the  installing  team  is  typically  much  more  familiar  with  the  systems  to  be  installed 
and  tested.  The  integration  and  testing  that  occurs  at  each  SSCC  TIF  that  was  previously 
discussed,  greatly  reduces  the  install  time  as  well  as  the  troubleshooting  required  within 
the  industrial  environment  at  the  shipyard.  Using  the  full  Turnkey  Approach  for 
installation  allows  for  radio  control  room  installation  later  in  the  ship  production 
schedule.  This  allows  PEO  C4I  SC  to  deliver  an  RCS  that  is  closer  to  “state  of  the  art” 
when  compared  to  other  approaches,  at  platform  delivery. 

Shipyard  Support  (SS):  A  weighting  factor  of  1.2  was  applied  to  shipyard 
support  based  on  the  agreed  amount  of  value  added  to  dollars  spent  within  this  category. 
SSCC  places  civil  service  and/or  contractor  personnel  on-site  at  the  shipyards  that  are 
skillfully  versed  in  the  various  Turnkey  processes.  They  are  in  place  to  interface  daily 
with  the  shipbuilder  directly.  The  on-site  personnel  improve  communication  between 
PEO  C4I  SC  and  the  shipbuilder,  which  allows  for  quicker  identification  and  resolution 
of  issues  and  concerns.  Major  issues  can  lead  to  stop  work  orders,  which  can  be 
extremely  expensive.  The  use  of  shipyard  support  personnel  helps  significantly  in 
avoiding  such  work  stoppages. 

5.  4th  and  Final  Iteration 

The  fourth  and  final  iteration  of  what  had  now  become  the  Jones,  Bigham,  and 
Goudreau  NROI  formula  to  be  used  for  project  analysis  was  determined  during  a  VTC 
with  Travis  Tillman  on  October  1,  2004.  The  final  version  is  shown  below: 


(WBS  -  Rework)  x  VAF  +  Avoidance  Costs 

NROI  %  =  - - - - 

Radio  Control  Suite  Program  Costs 


X100 
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Shown  next  is  the  final  iteration  with  the  WBS  categories  broken  out.  Note  that  in  the 
interest  of  space,  the  WBS  categories  are  presented  in  their  net  benefit  format  with 
rework  already  removed: 


NROI  (%) 


PM  x  0.1  +  ENG  x  0.4  +  SM  x  0.15  +  PR  x  0.4  +  INST  x  0.3  +  SS  x  0.2  +  MM  x  0.1  +  Avoidance  Costs 

- x  100 

Radio  Control  Suite  Program  Costs 


There  are  two  notable  differences  from  the  third  iteration.  First  the  VAFs  are 
shown  in  their  net  benefit  form.  Consequently,  one  is  no  longer  subtracted  from  the 
fractional  portion  of  the  formula.  This  allows  the  numerator  of  the  NROI  formula  to 
represent  the  net  benefit  to  the  project  similar  to  the  use  of  net  income  in  a  traditional 
private  sector  ROI  formulation.  Secondly,  rework  costs  are  no  longer  subtracted 
separately  in  the  numerator.  Instead,  rework  costs  are  subtracted  directly  within  their 
respective  categories.  This  gives  absolutely  no  benefit  to  rework  costs  determined  to  be 
required  as  a  result  of  PEO  C4I  SC.  In  the  third  iteration,  rework  costs  were  deducted 
separately  and  may  have  inadvertently  improved  the  fonnula  result.  Although  they  were 
also  subtracted  out  in  the  numerator,  this  was  only  after  being  multiplied  by  their 
respective  VAF.  Thus,  $10  of  rework  engineering  would  have  been  multiplied  by  1.4 
first  and  then  $10  subtracted  from  that  product.  This  would  have  given  a  credit  of  $4  to 
the  project  when  in  fact  the  rework  costs  should  provide  no  enhancement  to  the  PEO  C4I 
SC  NROI. 

The  fourth  iteration  of  the  fonnula  to  measure  PEO  C4I  SC  NROI  for  the  T-AKE 
project  best  meets  the  nine  criteria  previously  discussed.  First,  the  model  developed  for 
the  NROI  is  simple  and  theoretically  sound.  It  is  free  of  any  complex  mathematical 
equations  and  mimics  a  commonly  used  private  sector  return  on  asset  approach.  More 
importantly,  it  is  intuitive  such  that  with  brief  introduction  one  can  understand  how  it 
might  represent  a  ROI  for  dollars  invested  in  the  T-AKE  project. 

The  fourth  iteration  is  also  easy  to  implement.  The  data  required  to  consider  all 
program  costs  are  already  kept  by  platfonn  managers  at  PEO  C4I  SC.  There  is  some 
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manipulation  required  to  determine  rework  dollars  spent,  avoidance  costs,  overhead 
costs,  and  to  pull  travel  costs  from  each  of  their  respective  categories.  This  is  done  since 
dollars  spent  on  travel  costs  add  no  value  to  the  T-AKE  RCS,  whether  the  travel  is 
necessary  or  not.  However,  the  additional  time  required  by  the  platform  manager  to 
produce  the  data  required  is  still  relatively  minimal. 

The  final  NROI  formula  meets  the  credibility  standards  outlined  earlier  and 
results  are  also  able  to  be  replicated.  The  cost  data  used  in  the  seven  WBS  categories  are 
objective  numbers  taken  directly  from  the  platfonn  manager’s  cost  reports.  This 
maximum  use  of  objective  data  ensures  NROI  output  can  be  replicated.  The  VAFs  do 
add  subjectivity  to  the  formula,  but  efforts  were  taken  to  minimize  this  impact.  Platform 
managers  from  four  different  ship  classes  worked  together,  reaching  consensus  on  the 
actual  values  used  for  the  VAFs  in  each  of  the  seven  categories.  In  this  way  subjectivity 
was  minimized.  Furthermore,  this  also  enhances  credibility  since  personnel  that  will  be 
tasked  with  the  formula’s  implementation  were  included  in  its  development. 

This  formula  provides  the  utility  to  measure  NROI  for  past,  present,  and  future 
project  evaluation  as  well  as  cross-class  comparisons  of  similar  scope.  The  fourth 
iteration  enables  RCS  NROI  calculation  across  four  ship  classes  since  the  same  VAFs 
remain  applicable.  This  improves  fonnula  flexibility. 

The  fourth  iteration  is  also  comprehensive  in  nature  and  considers  all  of  the 
various  program  costs.  The  WBS  breaks  down  all  dollars  spent  on  the  RCS  into 
individual  categories.  All  of  these  are  considered  when  determining  value  added  to  the 
RCS.  All  relevant  program  costs  have  also  been  considered,  to  include  non-value  added 
costs  such  as  the  service  center  fee,  travel,  material,  and  rework. 

Finally,  careful  consideration  has  been  given  to  each  of  the  elements  included  in 
both  the  numerator  and  denominator  in  the  fourth  iteration.  Specifically,  their  individual 
and  combined  effects  on  the  overall  NROI  output  have  been  verified  to  be  in  accordance 
with  expectations.  For  example,  additional  non-value  added  costs  to  a  program  included 
in  the  formula  have  been  verified  to  actually  reduce  the  NROI.  This  meets  the  validity 
requirement  discussed  earlier  and  will  be  demonstrated  in  the  following  pages. 
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V.  FINAL  NOTIONAL  ROI  FORMULA 


A.  T-AKE  1  NROI 

The  final  NROI  formula  was  developed  to  be  applied  practically  and  represents 
the  most  appropriate  ROI  developed  for  and  adapted  to  the  design,  equipment 
procurement,  testing,  installation,  personnel  training,  and  certification  of  the  radio  room 
equipment  and  associated  gear  for  the  first  ship  of  T-AKE  new  construction.  The 
calculation  of  the  T-AKE  1  RCS  NROI  as  of  August  27,  2004  and  the  projected  overall 
NROI  are  provided  and  analyzed  below. 

Actual  costs  collected  through  August  27,  2004  and  projected  totals  are  shown  in 
Table  7  below  for  each  of  the  seven  WBS  categories.  Note  that  nothing  has  been  spent  as 
of  August  27th,  on  either  Installation  or  Shipyard  Support  since  none  has  been  required 
up  to  this  point  in  the  program. 


Table  7.  T-AKE  1  WBS  Category  Cost  Data 


WBS  Category  and  VAF 

Actual  Costs  through 

August  27,  2004 

Projected  Totals 

Program  Management  (0.2) 

$549,432 

$765,932 

Engineering  (0.4) 

$1,411,618 

$1,451,618 

Support  Management  (0.15) 

$157,897 

$242,897 

Production  (0.4) 

$469,349 

$869,349 

Installation  (0.3) 

$0 

$650,000 

Shipyard  Support  (0.2) 

$0 

$75,000 

Material  Management  (0.1) 

$101,899 

$191,899 

Table  8  shows  actual  costs  through  August  27,  2004  and  projected  totals  for  the 


T-AKE  RCS  project  for  non-value  added  costs  associated  with  the  program.  The  service 
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center  fee  represents  the  share  of  corporate  costs  or  overhead  charged  to  all  SPAWAR 
projects.  Travel  costs  are  normally  charged  to  the  individual  WBS  categories.  However, 
they  are  separated  from  their  respective  WBS  categories  since  they  are  a  non-value  added 
cost  to  the  T-AKE  1  ship.  The  role  of  PEO  C4I  SC  in  material  procurement  is  like  that  of 
a  middle  man,  buying  existing  equipment  from  suppliers  and  delivering  them  to  the 
buyer.  Therefore,  the  cost  of  the  material  itself  is  considered  a  non-value  added  cost. 
PEO  C4I  SC  adds  value  to  materials  only  after  installation,  testing  and  integration  in  the 
radio  room. 


Table  8.  T-AKE  1  Non-value  Added  Actual  Costs  and  Projected  Totals 


Non-Value  Added  Cost 

Actual  Costs  through 

August  27,  2004 

Projected  Totals 

Travel 

$121,119 

$181,119 

Service  Center  Fee 

$39,204 

$64,204 

Material 

$1,487,840 

$1,912,840 

Engineering  Rework  (must  be  subtracted  from 

ENG  category  prior  to  VAF  modification) 

$5,600 

$5,600 

Due  to  lack  of  specific  data  collection,  PEO  C4I  SC  generated  rework  was 
estimated  to  be  approximately  $5,600  in  the  engineering  category.  Also,  no  avoidance 
costs  were  tracked  for  the  T-AKE  1  RCS  project.  They  are  assumed  to  be  zero  in  an 
effort  to  be  conservative.  As  discussed  earlier,  rework  is  simply  removed  from  the  WBS 
category  prior  to  modification  by  the  VAF;  it  is  included  in  the  program  costs.  All  the 
data  have  now  been  presented  to  allow  calculation  of  an  NROI  for  the  T-AKE  1  RCS 
project  as  of  August  27,  2004  and  an  NROI  based  on  projected  totals.  These  calculations 
are  shown  below.  The  actual  NROI  as  of  August  27,  2004  is  calculated  as  follows: 


42 


^  tt\ _ vt  a  /  (WBS  -  Rework)  x  VAF  +  Avoidance  Costs 

NROI  %  =  - - - - 

T  -  AKE 1 RCS  Program  Costs 


X100 


PM  x  0.2  +  (ENG- Rework)  x  0.4  +  SM  x  0. 1 5  +  PR  x  0.4  +  INSTx0.3  +  SSx0.2  +  MM  x  0. 1 + AvoidanceCosts 

T  -  AKE  1  RCS  ProgramCosts 


$893,907  +  0 
$4,338,358 


X  100  =  20.6% 


The  projected  NROI  at  completion  of  the  T-AKE  1  RCS  program  is  calculated  as 
follows: 


$1,344,957  +  0 
$6,404,958 


X 100  =  2 1.0% 


More  detailed  calculations  are  provided  in  Table  9  on  the  next  page.  Notice  that  the 
actual  and  projected  NROI  remains  relatively  stable.  Of  course  this  assumes  that  no 
more  PEO  C4I  SC  generated  rework  will  be  incurred  and  avoidance  costs  will  continue  to 
be  zero.  The  slight  differences  in  NROI  are  due  to  a  shifting  in  proportions  between  the 
WBS  categories  as  the  project  matures. 


X100 
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Table  9 


T-AKE  1  Cost  Schedule  and  NROI  Determination 
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B.  NROI  COMPARABILITY  FOR  ONE  SHIP  CLASS 

The  developed  NROI  formula  possesses  the  flexibility  to  enable  calculation  of  the 
NROI  at  various  times  for  any  given  T-AKE  ship  RCS  project.  An  NROI  could  be 
estimated  prior  to  project  initiation  based  on  projected  WBS  and  associated  non-value 
added  costs.  It  can  be  calculated  at  any  time  during  the  project  cycle  with  actual  costs  up 
to  a  given  date.  NROI  is  also  able  to  be  calculated  at  project  completion.  If  senior  level 
buy  in  for  the  proposed  formula  was  obtained,  this  could  be  used  as  one  of  the 
benchmarks  to  indicate  the  amount  of  output  PEO  C4I  SC  is  providing  for  the  program 
dollars  invested.  A  priority  of  the  CNO  discussed  earlier,  this  would  begin  to  answer  the 
request  of  linking  actual  outputs  to  dollar  investments. 

Comparisons  made  of  the  T-AKE  RCS  project  NROI  at  different  times  during  the 
acquisition  cycle  must  be  carefully  considered.  The  underlying  assumptions  and 
circumstances  must  be  evaluated  cautiously  since  several  factors  can  attribute  to  NROI 
increasing  or  decreasing  at  discrete  points  in  the  project  cycle.  These  factors  include 
changes  in  the  amount  of  PEO  C4I  SC  generated  rework,  a  shift  in  the  proportional 
amounts  of  value  added  costs  in  the  WBS  categories,  adjustments  made  in  avoidance 
costs,  or  changes  in  the  amount  of  non-value  added  costs  attributed  to  the  project. 

PEO  C4I  SC  generated  rework  costs  are  subtracted  from  the  numerator  of  the 
NROI  formula  since  they  provide  no  net  benefit.  They  are  included  as  a  program  cost  of 
the  project  in  the  denominator.  Therefore,  as  the  amount  of  PEO  C4I  SC  generated 
rework  increases  over  the  project  cycle,  a  negative  impact  on  the  NROI  calculated  will 
occur.  This  is  one  possible  explanation  for  a  decreasing  NROI  over  the  life  of  a  project. 

Shifts  in  the  proportional  amounts  of  the  value  added  costs  in  the  WBS  categories 
over  the  life  of  the  project  will  also  affect  the  NROI.  This  may  be  due  to  no  fault  of  the 
program  itself.  For  example,  one  of  the  highest  value  added  categories  is  engineering. 
Table  7  from  the  previous  section  illustrates  how  as  of  August  27,  2004  most  of  the 
engineering  dollars  were  spent  while  some  lower  value  added  categories  such  as  shipyard 
support  and  installation  remained  unspent.  This  shift  in  WBS  category  costs  is  a  natural 
result  of  the  T-AKE  project  progression.  Significant  engineering  dollars  are  spent  up 
front  to  design  and  develop  the  RCS  while  installation  takes  place  much  later  in  the  cycle 
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after  most  engineering  dollars  have  been  spent.  This  shifting  of  WBS  category  costs 
could  be  responsible  for  changing  the  NROI  between  two  discrete  points  in  the  project 
cycle  since  each  WBS  category  is  converted  to  value  added  using  numerically  different 
factors.  A  non-proportional  shift  in  WBS  costs  to  lower  value  added  categories  will 
lower  the  NROI.  Conversely,  a  non-proportional  shift  in  WBS  costs  to  higher  value 
added  categories  will  have  the  effect  of  increasing  the  project  NROI. 

Costs  avoided  by  PEO  C4I  SC  during  the  life  of  the  T-AKE  project  will  have  a 
positive  effect  on  NROI.  Avoidance  costs  are  added  in  their  entirety  to  the  numerator 
and  are  not  included  in  the  denominator.  This  is  also  a  possible  explanation  for 
increasing  NROI  over  the  life  of  the  project.  Unfortunately,  no  avoided  costs  were 
captured  by  the  current  method  of  data  collection  used  for  this  project. 

Varying  amounts  of  non-value  added  costs  over  the  course  of  the  T-AKE  project 
will  have  a  direct  effect  on  the  project’s  NROI.  Non-value  added  costs  include  expenses 
such  as  travel,  overhead,  and  purchased  material.  These  costs  are  part  of  the  NROI 
formula’s  denominator  but  are  not  included  in  the  numerator  since  they  provide  no  “PEO 
C4I  SC  added  value”  to  the  radio  room.  Therefore,  changing  amounts  of  non-value 
added  costs  will  impact  the  NROI  between  any  two  points  in  the  project  cycle. 

NROI  between  different  ships  of  the  T-AKE  class  can  also  be  evaluated.  For 
example,  the  NROI  of  the  T-AKE  1  RCS  can  be  compared  to  the  NROI  of  the  successive 
T-AKE  ship  radio  rooms  as  the  actual  or  projected  cost  data  become  available.  This 
could  potentially  indicate  the  ROI  for  employing  PEO  C4I  SC  for  multiple  ships  within 
the  T-AKE  ship  class  or  indicate  problems  with  a  specific  T-AKE  hull.  As  with  the 
comparisons  made  at  different  points  in  one  T-AKE  RCS  project  cycle,  comparisons 
made  between  successive  T-AKE  radio  rooms  must  also  be  evaluated  carefully.  The 
potential  causes  for  changing  NROI  analyzed  and  described  previously  for  a  single  T- 
AKE  project  all  apply  to  comparisons  made  between  different  T-AKE  hulls.  However, 
additional  considerations  must  be  made  when  making  comparisons  between  hulls.  The 
natural  learning  curve  between  successive  identical  platforms  and  the  significant  potential 
for  increased  avoidance  costs  should  also  be  assessed. 
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A  learning  curve  naturally  occurs  when  repetitive  tasks  are  performed  by  an 
organization  such  as  PEO  C4I  SC.  This  will  occur  for  many  reasons  including  increased 
familiarization  with  the  tasks  required  and  improved  project  coordination.  The  DoD  not 
only  considers  this,  but  expects  it  as  well,  when  awarding  contracts  for  multiple  item 
procurements  such  as  successive  ship  hulls  (Rendon,  2004).  Improved  efficiencies  will 
be  realized  and  some  costs  from  the  first  RCS  will  be  avoided  on  subsequent  platforms. 
For  example,  much  of  the  engineering  design  and  development  costs  will  be  avoided  on 
later  T-AKE  hulls  since  they  will  be  “cookie  cutter”  examples  of  the  first.  Also,  it  is 
likely  that  SPA  WAR  generated  rework  on  a  previous  T-AKE  hull  will  be  avoided  on 
later  T-AKE  hulls  due  to  lessons  learned.  This  improved  efficiency  and  avoided  costs 
can  affect  the  NROI  either  positively  or  negatively.  Increased  avoidance  costs  for 
successive  hulls  would  increase  the  NROI  from  one  hull  to  the  next.  Conversely, 
significantly  reducing  engineering  costs  in  later  hulls  may  have  a  negative  effect  on 
NROI  since  this  is  one  of  the  two  highest  value  added  WBS  categories. 

It  is  essential  to  understand  that  any  ROI  formulation,  in  and  of  itself,  has  very 
little  meaning.  It  requires  a  comparison  to  some  relatively  similar  baseline  or  benchmark 
to  gain  significance.  It  is  also  important  that  when  making  these  comparisons,  the 
context  of  the  two  numbers  being  compared  is  well  understood.  This  is  no  different  than 
when  considering  traditional  ROI.  Consider  an  investor  who  gained  five  percent  on  his 
or  her  portfolio  over  the  previous  year.  For  this  to  be  considered  a  respectable  ROI 
would  depend  on  several  factors.  If  this  was  a  low  risk  investment,  five  percent  may  be 
adequate.  If  this  were  an  aggressive  investment,  it  is  likely  that  five  percent  would  be 
considered  sub  par.  However,  if  the  economy  was  in  a  recession,  five  percent  return  may 
be  superior  regardless  of  the  investment  profile.  The  point  is  that  an  NROI  requires  a 
comparison  data  set  to  give  it  meaning.  Thus,  careful  consideration  must  be  given  to  the 
context  of  two  NROI  being  compared  to  fully  understand  their  significance,  just  as  with 
comparing  traditional  ROI  in  the  private  sector. 
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c. 


LPD-19  NROI 


Figure  2.  LPD-17  Class  (Global  Security,  2004) 


LPD-19  is  the  third  vessel  of  the  new  LPD-17  ship  class.  Data  were  collected  in 
the  same  WBS  category  format  to  allow  calculation  of  an  NROI  in  a  similar  manner  that 
the  T-AKE  NROI  was  ascertained.  Recall  that  the  same  VAFs  used  for  the  T-AKE  RCS 
are  also  valid  for  the  LPD-17  class.  The  details  of  the  NROI  calculations  for  LPD-19  are 
based  on  actual  cost  data  through  September  4,  2004  and  on  projected  totals.  WBS 
category  costs  are  shown  in  Table  10. 

Table  10.  LPD-19  WBS  Category  Cost  Data 


WBS  Category  and  VAF 

Actual  Costs  through 

September  4,  2004 

Projected  Totals 

Program  Management  (0.2) 

$1,352,275 

$1,985,275 

Engineering  (0.4) 

$195,000 

$612,000 

Support  Management  (0.15) 

$1,096,160 

$1,328,260 

Production  (0.4) 

$3,128,951 

$6,252,951 

Installation  (0.3) 

$1,293,766 

$3,086,509 

Shipyard  Support  (0.2) 

$0 

$0 

Material  Management  (0.1) 

$1,186,544 

$2,373,088 
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The  value  added  costs  represent  those  spent  for  the  RCS  of  the  LPD-19  to  allow 
better  comparability  to  the  T-AKE  results.  Non-value  added  costs  as  of  September  4, 
2004  and  projected  totals  are  shown  in  Table  11.  Note  that  to  date  no  SPAWAR 
generated  rework  has  been  incurred  for  LPD-19.  This  is  most  likely  due  to  the  fact  that 
she  is  the  third  ship  of  her  class  and  many  lessons  have  already  been  learned  whereas  the 
previous  data  were  for  the  inaugural  ship  of  the  T-AKE  class.  In  an  effort  to  remain 
conservative,  avoidance  costs  for  LPD-19  are  also  assumed  to  be  zero  since  they  were  not 
tracked. 

Table  11.  LPD-19  Non-value  Added  Actual  Costs  and  Projected  Totals 


Non-Value  Added  Cost 

Actual  Costs  through 

September  4,  2004 

Projected  Totals 

Travel 

$71,103 

$229,353 

Service  Center  Fee 

$85,433 

$117,083 

Material 

$2,059,481 

$2,271,481 

Engineering  Rework 

$0 

$0 

The  actual  NROI  for  the  LPD-19  as  of  September  4,  2004  is  calculated  as 
follows: 


(WBS  -  Rework)  x  VAF  +  Avoidance  Costs 

NROI  %  =  - - - - 

LPD  - 1 9  RCS  Program  Costs 


X100 


PM 


x  0.2  +  ENG  x  0.4  +  SM  x  0.15  +  PR  x  0.4  +  INST  x  0.3  +  SS  x  0.2  +  MM  x  0.1  +  Avoid  Costs 

LPD  -19  RCS  Program  Costs 


X  100 


$2,271,244  +  0 
$10,468,713 


X 100  =  2 1.7% 
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The  projected  NROI  at  completion  of  the  LPD-19  RCS  program  is  calculated  similarly  as 
follows: 


$4,503,296  +  0 
$18,256,000 


X 100  =  24.67% 


The  NROI  of  LPD-19  does  increase  over  the  life  of  the  project.  This  is  primarily  due  to 
the  significant  number  of  dollars  still  left  to  be  spent  on  the  high  value-added  categories 
of  production  and  installation.  More  detailed  calculations  are  provided  in  Table  12  on  the 
next  page 
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Table  12. 


LPD-19  Cost  Schedule  and  NROI  Determination 
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D.  NROI  COMPARABILITY  FOR  MULTIPLE  SHIP  CLASSES 


NROI  comparisons  between  PEO  C4I  SC  installed  radio  control  rooms  can  also 
be  made  between  ship  classes.  Some  of  the  same  factors  that  must  be  considered  when 
making  comparisons  between  radio  rooms  of  the  same  hull  must  also  be  considered  here. 
Differences  in  avoidance  costs  realized,  the  proportions  of  costs  within  the  WBS  cost 
categories,  and  the  amount  of  PEO  C4I  SC  generated  rework  and  other  non-value  added 
costs  will  all  promote  differences  in  the  two  NROI.  However,  there  is  the  potential  for 
these  factors  to  have  a  much  greater  effect  when  comparing  NROI  of  radio  rooms 
between  different  ship  classes.  Differences  in  the  learning  curve,  the  relative  amounts  of 
value  added  costs  and  non-value  added  costs,  and  the  VAFs  themselves  must  all  be 
considered  when  making  comparisons  between  different  ship  classes. 

The  learning  curve  will  vary  significantly  between  hulls  from  different  ship 
classes  for  several  reasons.  First,  technical  complexity  between  the  radio  control  rooms 
of  different  ship  classes  can  vary  widely.  The  RCS  installed  in  a  T-AKE  class  vessel 
would  not  be  considered  very  complex  when  compared  to  the  RCS  of  an  aircraft  carrier. 
Increased  complexity  can  either  increase  or  decrease  the  learning  curve.  Also,  the 
number  of  hulls  being  built  within  a  given  ship  class  will  affect  the  amount  of  benefit 
realized  from  a  learning  curve.  If  only  a  few  hulls  are  to  be  constructed  as  compared  to  a 
ship  class  with  a  large  number  of  hulls  designated,  there  will  be  less  benefit  from  the 
learning  curve  achieved  for  the  smaller  ship  class.  The  different  learning  curves  may 
affect  the  amount  of  cost  avoidance  realized  from  hull  to  hull  and,  thus  affect  the  NROI. 

The  relative  amounts  of  value  added  costs  between  the  WBS  categories  will  also 
produce  differences  in  NROI  compared  between  ship  classes.  A  warship  on  the  cutting 
edge  of  technology  would  require  larger  amounts  of  dollars  to  be  spent  in  the 
engineering,  production  (integration),  and  installation  categories  than  an  auxiliary  ship 
with  average  radio  control  room  technology.  These  shifts  affect  the  net  benefit,  or 
numerator,  detennined  in  the  NROI  fonnula.  Consequently,  shifts  in  WBS  costs  between 
ship  classes  will  affect  the  resulting  NROI. 

The  amount  of  non-value  added  costs  may  vary  considerably  between  hulls  of 
different  ship  classes.  The  amount  of  material  required  for  purchase  to  support  the  radio 
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room,  the  amount  of  travel  required,  and  the  service  center  fee  charged  will  all  affect  the 
NROI.  Most  notably  is  the  amount  of  material  that  needs  to  be  acquired.  The  CVN-68 
class  will  require  appreciably  more  material  to  be  purchased  than  an  auxiliary  ship  like 
the  T-AKE  class.  This  large  difference  in  non-value  added  costs  due  to  material 
differences  will  dampen  the  NROI  for  the  carrier  since  the  value  of  the  denominator  will 
increase  with  no  increase  in  the  numerator.  The  amount  of  travel  required  may  also  vary 
considerably  depending  on  the  location  of  the  shipyard  chosen  to  build  a  ship  class. 
Also,  depending  on  the  allocation  method  for  overhead,  the  amount  of  the  service  center 
fee  may  differ  considerably  between  ship  classes.  The  different  relative  amounts  of  non¬ 
value  costs  may  have  a  large  impact  on  the  NROI  when  comparing  hulls  from  different 
ship  classes. 

Finally,  the  VAFs  used  to  determine  an  NROI  for  a  given  ship  class  can  also 
differ.  For  the  context  of  this  thesis,  the  VAFs  chosen  were  developed  by  SSCC  resident 
experts  from  the  T-AKE,  CVN-68,  LPD-17,  and  LHD-8  classes  of  ships.  These  VAFs 
represent  their  best  judgment  as  to  the  value  PEO  C4I  SC  adds  to  dollars  spent  within  the 
respective  WBS  categories.  However,  the  VAFs  would  likely  not  be  the  same  for  a 
Virginia  class  submarine.  It  is  relatively  easy  to  surmise  that  such  differences  would 
change  the  bounds  of  the  equation  entirely. 

NROI  comparisons  made  between  hulls  of  different  ship  classes  must  be  made 
cautiously.  The  learning  curve,  relative  amounts  of  value  added  costs  and  non-value 
added  costs,  and  the  NROI  VAFs  can  all  differ  appreciably.  This  may  skew  the  results 
to  indicate  one  project  is  operating  better  than  the  other  when  in  fact  some  of  the 
differences  may  be  attributable  to  the  uniqueness  of  the  project  itself.  Let  us  consider 
again  comparisons  of  traditional  ROI  measurements.  The  return  in  the  bond  market  is 
not  expected  to  equal  returns  in  the  stock  market.  They  are  two  different  investment 
vehicles.  However,  the  expected  return  on  bonds  can  be  approximated  if  the  return  on 
the  stock  market  is  known  once  an  adjustment  is  made  to  take  into  account  the  different 
investment  options.  This  adjustment,  commonly  called  a  risk  premium,  is  based  on 
market  history.  A  similar  type  of  adjustment,  once  determined,  could  be  used  to  compare 
NROI  of  entirely  different  ship  classes. 
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E.  COMPARISON  OF  RCS  NROI:  T-AKE  1  AND  LPD-19 

The  LPD-19  and  T-AKE  1  RCS  NROI  both  surpass  twenty  percent.  This 
indicates  over  a  twenty  percent  ROI  based  on  the  NROI  formula  developed.  On  the 
surface,  both  projects  appear  to  add  significant  value  to  their  respective  radio  rooms. 
However,  it  is  important  to  consider  the  frame  of  reference  used  when  making  this 
judgment.  A  twenty  percent  return  in  the  stock  market  by  most  benchmarks  is  excellent. 
Conversely,  to  more  accurately  evaluate  the  value  added  to  a  radio  room,  a  new 
benchmark  must  be  created  based  on  similar  data  collected  over  time. 

The  LPD-19  RCS  project  has  both  a  slightly  higher  NROI  to  date  and  at  project 
completion  based  on  projected  totals.  Taken  at  face  value,  this  would  indicate  that  the 
LPD-19  project  funds  are  better  spent  since  dollar  for  dollar  they  add  more  value  than  the 
funds  spent  on  the  T-AKE  1  venture.  However,  as  indicated  previously,  NROI  results 
cannot  be  compared  without  taking  into  consideration  other  factors.  One  must  consider 
the  complex  underlying  circumstances  for  derivation  of  the  numbers  to  make  a  fair 
comparison.  Best  use  of  the  NROI  is  between  ships  of  the  same  class  vice  between 
classes.  However,  additional  refinement  of  the  methodology  may  make  such  comparison 
between  platforms  more  valid.  We  remind  readers  that  ROI  in  itself  is  but  one  piece 
among  many  types  of  financial  infonnation  available  to  platform,  program  and  financial 
management  decision  makers  when  comparing  actual  and  potential  investment  options. 

The  proportions  of  dollars  spent  within  the  WBS  categories  for  the  two  ships  vary 
considerably.  Table  13  on  the  following  page  shows  a  breakdown  of  each  individual 
WBS  category  cost  as  percentage  of  total  WBS  dollars  projected  to  be  spent.  Most  of 
these  differences  are  inherent  to  the  project.  The  largest  difference  is  in  the  engineering 
WBS  category.  A  much  larger  proportion  of  WBS  dollars  are  spent  on  engineering  for 
the  T-AKE  1  RCS  than  on  LPD-19.  This  is  intuitive  since  significantly  more  engineering 
dollars  are  required  for  the  first  ship  of  any  class.  This  is  the  case  with  the  T-AKE- 1 
RCS.  LPD-19  is  the  third  ship  of  the  LPD-19  class,  and  consequently,  requires  less 
expenditure  in  the  engineering  WBS  category. 
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Table  13.  Individual  WBS  Totals  as  a  Percentage  of  Total  Projected  WBS  Costs 


WBS  Category 

T-AKE  1  RCS  (%) 

LPD-19  RCS  (%) 

Program  Management 

18 

13 

Engineering 

34 

4 

Support  Management 

6 

8 

Production 

20 

40 

Installation 

15 

20 

Shipyard  Support 

2 

0 

Material  Management 

5 

15 

The  LPD-19  RCS  requires  considerably  more  dollars  to  be  spent  in  the  production 
WBS  category.  This  includes  both  absolute  dollars  and  as  a  percentage  of  total  WBS 
category  costs.  This  also  is  to  be  expected  since  the  LPD-19  RCS  project  is 
approximately  three  times  the  size  of  the  T-AKE  1  RCS  project  based  on  total  program 
dollars.  The  LPD-19  RCS  is  more  complex  in  nature  than  the  RCS  of  the  T-AKE  1 
vessel.  Production  is  a  high  value  added  WBS  category  and  this  difference  is  the  leading 
contributor  to  LPD-19’s  higher  NROI.  Thus,  while  LPD-19  does  have  a  slightly  higher 
NROI,  this  is  due  largely  to  the  inherent  differences  between  the  programs. 

LPD-19  spends  no  dollars  in  the  shipyard  support  category.  This  is  not  a  large 
contributor  to  the  NROI  differences.  However,  it  is  important  to  re-emphasize  that  when 
comparing  ships  from  different  classes  some  of  the  VAFs  chosen  may  be  different  or  not 
apply  at  all,  as  in  this  case.  This  must  be  carefully  considered  when  making  these 
evaluations. 

The  comparison  of  non-value  added  costs  is  shown  in  Table  14.  Note  that  in  all 
cases  except  for  rework,  LPD-19  has  a  larger  burden  of  non-value  added  costs.  This  is 
expected  for  both  the  material  required  and  the  service  center  fee  considering  the  larger 
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scope  of  the  project.  Travel  is  a  separate  function  of  the  chosen  location  to  build  the  ship. 
LPD-19  should  also  be  expected  to  have  less  rework  costs  given  that  it  is  the  third  ship  of 
the  LPD-17  class. 


Table  14.  Comparison  of  Total  Projected  Non-value  Added  Costs 


Non-value  Added  Cost 

T-AKE  1  RCS  ($) 

LPD-19  RCS  ($) 

Material 

1,912,840 

2,271,481 

Service  Center  Fee 

64,204 

117,083 

Travel 

181,119 

229,353 

Estimated  Rework 

5,600 

0 

One  aspect  not  indicated  in  the  numbers  is  that  an  integral  system  required  for 
each  RCS  is  procured  in  an  entirely  different  manner.  The  HF  system  for  the  T-AKE  1, 
due  to  its  uniqueness,  was  procured  by  the  RCS  manager  using  program  dollars  allocated 
to  the  T-AKE  1  RCS  program.  However,  the  HF  system  for  the  LPD-19  was  procured 
using  dollars  from  a  separate  pot  of  money  other  than  from  the  LPD-19  RCS  program. 
The  different  methods  of  procurement  and  associated  accounting  for  program  dollars  do 
affect  the  projects  NROI  as  the  formula  stands.  Since  material  is  treated  as  a  non-value 
added  cost,  increases  in  dollars  spent  on  material  will  lower  a  project’s  NROI.  Therefore, 
the  effect  is  to  lower  the  T-AKE ’s  NROI  and,  subsequently,  increase  the  LPD-19’s 
NROI.  Given  that  the  HF  system  can  cost  in  upwards  of  millions  of  dollars,  the  effect  on 
these  NROI  is  not  trivial.  For  better  comparison,  this  adjustment  should  be  made. 

Finally,  while  avoidance  costs  were  not  tracked  due  to  the  timing  of  this 
endeavor,  they  could  have  potentially  had  a  major  impact  on  the  NROI  of  the  two 
projects.  For  example,  since  LPD-19  is  not  the  inaugural  vessel,  it  is  plausible  to 
presume  that  many  costs  were  avoided  on  the  third  ship  versus  those  that  were  incurred 
on  the  first.  Also,  since  it  is  likely  that  large  equipment  purchases  for  multiple  class  ships 
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result  in  a  cost  savings,  these  would  also  increase  the  avoidance  costs  credited  to  a 
particular  program.  These  avoidance  costs  would  positively  impact  the  project  NROI. 

Considering  the  number  of  inherent  differences  between  the  programs  and  the 
small  numerical  differences  between  the  NROI  calculated,  it  cannot  be  concluded  that  the 
differences  reflected  in  the  NROI  calculation  are  due  to  one  program  operating  more 
efficiently  than  the  other.  Unfortunately,  at  this  point  evenhanded  comparisons  made  of 
NROI  between  ship  classes  can  be  difficult,  depending  on  the  degree  of  scope  differences 
between  the  programs.  While  the  LPD-19  and  T-AKE  1  RCSs  both  use  the  same  VAFs, 
many  disparities  still  exist  between  the  programs  that  must  be  considered  to  level  the 
playing  field  between  the  projects.  This  comparison  might  further  be  complicated  if  the 
use  of  different  VAFs  were  required.  Continued  data  collection  and  NROI  calculations 
need  to  be  made  in  order  to  better  establish  an  expected  or  normal  benchmark  to  facilitate 
comparison  and  more  accurately  understand  how  differences  in  various  programs  can  be 
expected  to  affect  the  NROI.  Armed  with  this  knowledge,  comparisons  between  ship 
classes  can  be  more  easily  made. 


F.  CHALLENGES 

1.  Formula  Limitations 

There  are  several  challenges  ahead  for  PEO  C4I  SC  to  successfully  implement  the 
developed  NROI  process  and  formula  application  within  the  organization.  First,  the 
subjective  nature  of  the  VAFs  will  require  senior  level  management  buy-in  to  strengthen 
the  formula’s  credibility.  Phillips  and  Phillips  point  out  three  audiences  which  must  buy 
in  to  the  ROI  process  for  it  to  be  useful.  They  include  the  practitioners  who  are 
responsible  for  implementing  the  formula  and  held  accountable  by  the  results,  senior 
level  management  who  hold  the  practitioners  accountable  for  these  results,  and 
researchers  (Phillips  and  Phillips,  2002:  12).  The  VAFs  and  the  entire  NROI  formula 
were  developed  in  cooperation  with  several  experts  at  the  practitioner  level.  However,  in 
order  for  the  NROI  formula  to  satisfactorily  represent  program-performance,  senior 
managers  responsible  for  evaluating  this  performance  must  buy  in  to  the  formula  and, 
more  importantly,  the  selected  values  of  the  VAFs.  For  example,  senior  management 


57 


must  agree  that  an  engineering  dollar  spent  on  the  T-AKE  platfonn  RCS  produces 
approximately  one  dollar  and  forty  cents  worth  of  value.  The  NROI  fonnula  will  not  be 
used  by  the  practitioners  if  it  will  not  hold  weight  with  their  managers.  This  is  true  of 
any  new  process  initiatives  in  the  work  place. 

Segregation  of  responsibilities  with  regard  to  data  collection  and  evaluation  must 
also  be  achieved  to  strengthen  the  credibility  of  the  results.  In  our  case  study  due  to 
personnel  availability,  Travis  Tillman  provided  assistance  in  both  formula  development 
and  data  collection.  However,  as  Platform  Manager  he  is  also  responsible  for  the  results 
of  the  T-AKE  Class  RCS.  There  is  a  clear  motive  to  skew  the  results  if  the  person 
responsible  for  the  results  also  collects  and  interprets  the  data.  Segregation  of  these 
duties  is  necessary  going  forward  if  the  NROI  metric  is  to  be  utilized  within  PEO  C4I  SC 
for  actual  management  decision  making. 

Increased  resources  will  be  required  from  PEO  C4I  SC  to  modify  the  NROI 
development  process  and  collect  data  for  the  evaluation  of  other  projects.  No  two 
projects  within  the  PEO  C4I  SC  organization  are  exactly  alike  and  some  modification  of 
the  work  done  for  this  project  will  be  required.  This  will  demand  managerial  effort  to 
develop  an  approved  NROI  measurement  and  some  amount  of  resources  to  collect  the 
required  data  for  other  projects. 

Ideally,  the  NROI  process  should  be  implemented  at  the  inception  stage  of  a 
project  instead  of  during  or  after  the  project  is  in  progress.  The  T-AKE  RCS  program 
was  already  well  underway  before  this  research  began.  Consequently,  some  data 
collection  opportunities  were  missed  and  some  data  either  had  to  be  estimated  or 
discarded  altogether  in  an  effort  to  remain  conservative.  This  limits  the  accuracy  of  the 
formula.  For  example,  rework  costs  were  not  collected  up  to  the  point  the  formula  was 
created  and  had  to  be  estimated.  Avoidance  costs  were  also  not  being  tracked  and  were 
assumed  to  be  zero  since  including  an  artificial  number  would  have  increased  the  NROI. 

The  final  limitation  is  that  the  makeup  of  the  fonnula  itself  limits  the  possible 
range  of  results  more  than  a  traditional  ROI  formula  would.  Traditional  ROI  used  in  the 
private  sector  is  not  as  bound  by  the  formula  itself.  ROI  values  in  the  private  sector 
nonnally  range  from  negative  one  hundred  percent  (a  total  loss  which  could  also  be 
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surpassed  in  rare  cases)  to  an  unbound  positive  value.  Negative  and  positive  ROI  are 
routinely  observed  in  the  stock  market.  Any  organization  with  a  negative  net  income  will 
have  a  negative  ROI.  Organizations  with  a  positive  net  income  will  have  a  positive  ROI. 
However,  the  NROI  formula  developed  for  PEO  C4I  SC  prevents  a  negative  number  for 
a  result.  The  lowest  value  achievable  would  result  from  minimum  WBS  costs,  significant 
rework  costs,  and  no  avoidance  costs.  Theoretically,  this  could  be  zero,  although 
practically  it  would  have  to  be  some  number  above  zero  since  some  amount  of  work 
performed  will  not  be  classified  as  rework. 

The  values  chosen  for  the  VAFs  also  place  an  upper  bound  on  the  value  attainable 
for  a  NROI.  They  place  an  explicit  upper  bound  if  there  are  no  avoidance  costs. 
Consider  the  VAFs  developed  for  the  T-AKE  RCS  project.  The  largest  values  of  0.4  are 
from  the  WBS  categories  of  engineering  and  production.  The  highest  NROI  theoretically 
achievable  without  avoidance  costs  would  result  from  all  costs  placed  into  these  two 
categories  and  no  non-value  added  costs  like  rework,  overhead,  or  travel.  This  would 
place  an  upper  theoretical  limit  of  the  NROI  of  forty  percent.  Consequently,  the  VAFs 
chosen  for  a  particular  project  will  set  bounds  for  the  potential  results  derived  from  the 
NROI  formula.  This  limitation  must  be  considered  when  evaluating  the  results. 

2.  Applicability  Limitations 

The  NROI  formula  developed  for  the  T-AKE  RCS  has  clear  applicability 
limitations  to  other  DoD  and  public  sector  programs.  This  application  of  the  NROI 
formula  is,  by  itself,  relevant  only  to  the  platforms  for  which  the  VAFs  were  developed. 
These  include  radio  rooms  from  the  T-AKE,  LPD-17,  CVN-68,  and  LHD-8  ship  classes. 
These  factors  define  how  the  key  players  involved  in  their  development  numerically 
define  how  dollars  spent  within  the  different  WBS  categories  result  in  added  value  to 
their  respective  RCS  projects.  These  are  not  applicable  to  other  programs.  The  VAFs 
would  most  likely  change  if  they  were  developed  for  another  public  sector  organization 
since  their  personnel  may  view  the  process  of  adding  value  much  differently. 

The  use  of  WBS  categories  is  also  not  universal  across  the  DoD.  WBS  categories 
are  primarily  used  within  the  acquisition  community  to  allow  program  managers  and 
contractors  to  breakdown  large  acquisition  projects  into  smaller  more  manageable  tasks. 


59 


Therefore  the  use  of  these  WBS  categories  as  a  data  collection  basis  for  an  NROI 
application  is  limited  to  this  community. 

The  NROI  development  methodology  of  defining  how  an  organization  adds  value 
via  its  products  or  services  and  creating  a  representative  metric  is  not  unique  to  PEO  C4I 
SC.  The  methodology  employed  in  this  thesis  can  be  applied  to  any  public  sector 
organization  that  is  willing  to  go  through  the  necessary  steps  to  define  how  they  create 
value  in  the  products  or  services  they  provide.  Several  public  sector  organizations  have 
already  successfully  implemented  ROI  metrics  as  one  benchmark  to  demonstrate  their 
performance.  However,  due  to  the  distinct  public  sector  outputs,  this  endeavor  is  often 
times  challenging.  Private  sector  organizations  typically  have  the  common  goal  of 
producing  profit.  Conversely,  public  sector  organizations  have  a  myriad  of  different 
goals  most  of  which  do  not  include  the  generation  of  profit. 
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VI.  SUMMARY  AND  CONCLUSION 


A.  SUMMARY 

This  thesis  has  probed  the  efforts  of  diverse  institutions  in  the  quest  for  a 
workable  method  of  detennining  ROI  in  the  public  sector.  Past  efforts  provided  valuable 
lessons  learned  which  were  identified  and  discussed.  The  United  States  Postal  Service 
was  notably  successful,  but  perhaps  only  because  they  function  much  more  like  a  private 
company  than  is  the  norm  for  public  sector  entities.  In  Australia,  the  bold  effort  to 
develop  a  “value  added”  approach  within  the  entire  government  budget  process  made 
some  progress,  but  fell  short  primarily  due  to  a  focus  on  analysis  lacking  a  quantifiable 
formula  for  detennining  the  output.  The  Royal  New  Zealand  Navy  enjoyed  relative 
success  but  was  hampered  in  some  ways  by  an  inability  to  screen  out  other  influences 
besides  the  bonus  scheme  for  retention  of  MEs.  The  United  States  Navy  Dental 
community  effort  was  largely  successful  in  the  development  of  an  ROI  methodology,  but 
fell  short  in  their  ability  to  use  the  output  since  they  lacked  trust  in  the  method  developed 
for  determining  some  of  the  inputs. 

The  ultimate  point  of  this  thesis  was  to  develop  an  NROI  fonnula  specifically 
tailored  for  the  SPAWAR  PEO  C4I  SC  RCS.  The  sequential  process  of  NROI  formula 
development  for  the  T-AKE  RCS  was  reproduced  as  a  way  of  highlighting  the  significant 
challenges  inherent  in  this  effort  as  well  as  to  serve  as  a  roadmap  for  others  to  follow.  It 
can  only  be  hoped  that  use  of  this  roadmap  may  help  others  to  avoid  some  of  the 
distractions  and  obstacles  that  the  lessons  learned  exposed.  The  utility  of  the  final 
formula  was  gratifying  in  that  it  was  readily  usable  not  only  for  the  T-AKE  class  but  also 
as  a  comparison  tool  for  the  RCS  projects  from  LPD-17,  CVN-68,  and  LHD-8  ship 
classes.  Of  course  adjustments  must  be  created  to  directly  compare  ships  with  vastly 
different  asset  bases  in  play,  but  the  fonnulaic  approach  is  sound. 


B.  CONCLUSIONS 

The  initial  goal  of  this  effort  was  to  develop  a  formula  that  would  ultimately 
provide  DoD  with  a  means  of  incorporating  NROI  criteria  in  the  decision  making  process 
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from  the  highest  levels  of  government.  From  there  the  scale  was  refined  to  creation  of  an 
NROI  formula  specifically  tailored  for  SPAWAR  PEO  C4I  SC  leadership  to  utilize 
concerning  new  ship  construction.  An  additional  reduction  in  scope  was  soon 
forthcoming  with  the  new  focus  zeroed  in  on  the  T-AKE  1  RCS.  The  narrowing  of  scope 
from  first  intentions  to  ultimate  conclusion  was  necessitated  by  the  sheer  complexity  and 
inherent  resistance  to  this  effort.  The  relative  failure  to  successfully  evaluate  ROI  in  the 
public  sector  despite  past  efforts  is  well  known  and  has  served  to  create  an  indifference  to 
trying  among  many  civil  servants  and  military  personnel.  The  result  of  this  effort  should 
at  least  serve  to  embolden  those  who  would  continue  to  press  on  in  this  endeavor. 

Perhaps  the  most  important  lesson  learned  throughout  this  process  was  that  the 
scope  of  such  an  effort  must  remain  focused  to  be  successful.  Once  NROI  formulae  are 
established  at  the  project  level  and  successfully  demonstrated,  an  expansion  of  utilization 
could  be  envisioned  which  may  ultimately  lead  to  the  availability  of  ROI  data  for  senior 
DoD  decision  making.  The  goal  now  is  to  continue  moving  in  the  right  direction  to 
answer  the  CNO’s  call  to  action  for  developing  a  method  of  determining  ROI  to  assist  the 
Navy  decision  making  process.  The  CNO  reaffirmed  this  vision  in  a  visit  to  the  Naval 
Postgraduate  School  on  March  17,  2004.  In  the  words  of  Douglas  A.  Brook,  Dean  of  the 
Graduate  School  of  Business  and  Public  Policy  at  NPS: 

The  CNO  is  clearly  focused  on  the  business  management  aspects  of 
running  the  Navy.  He  is  challenging  the  Navy  to  focus  on  effectiveness 
and  costs  rather  than  what  he  perceives  to  have  been  a  "readiness  at  any 
cost"  approach  (Brook,  2004). 

Dean  Brook  also  pointed  out  that  one  of  the  specific  tasks  noted  by  the  CNO  for  the 
faculty  and  students  of  NPS  included  finding  “a  way  to  calculate  ROI.”  It  is  our  sincere 
desire  and  belief  that  we  have  assisted  in  attaining  this  goal  through  the  development,  test 
application  and  evaluation  of  the  NROI  methodology  presented  in  this  thesis. 
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